mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
fix: alert noise + loop stalls + kn-data-sync OOM from #alerts triage (v0.282.1) (#1210)
## Summary
Fixes for the open and noise groups from the 2026-10-22 #alerts triage.
**Noise that should not page:**
- `stripe_service.py`: "Invalid webhook signature" now logs WARNING. All
42 hits on 10-21 came from `facebookexternalhit` via Cloudflare with a
malformed header. Every real Stripe delivery since 10-14 got a 200.
(#alerts 1562375740685815953)
- `error_monitor/known_issues.py`: `startup_probe` also matches Cloud
Run's "Default STARTUP **TCP** probe failed" wording.
(1562224513276706933)
- `frontend/lib/crash-reporter.ts`: a `removeChild`/`insertBefore`
NotFoundError is benign **only while the page is machine-translated**,
i.e. `<html>` has `translated-ltr`/`translated-rtl`. In the 10-21 crash,
Chrome had translated `/en/app` into Spanish ("Crea un video karaoke" is
not our string), and the customer's job completed and was downloaded.
(1562541569238638725)
**Event-loop stalls** (continues the WIP freeze item in BACKLOG):
- `review.py` audio-editor routes: `apply_edit`, `get_metadata_from_gcs`
and the GCS `download_file`/`upload_file` calls now go through
`asyncio.to_thread`. They were stalling the loop about 10s per edit (job
330d796c, 10-19). (1561861805695967363)
- `internal.py`: `process_conversion_emails` is imported at module top.
The lazy import inside the hourly cron's `async def` wrote a `.pyc` on
the loop, stalling it up to 15.3s, 4× on 10-20/21. (1562224514933334047)
- `backend/Dockerfile`: `python -m compileall` over `/app/backend` and
`/app/karaoke_gen`, so no lazy import compiles at runtime.
- `test_no_blocking_async_routes.py`: `BLOCKING_CALLS` now includes
`apply_edit`, `get_metadata_from_gcs`, `upload_file` and
`download_file`. Two more routes in `review.py` were caught and wrapped:
the preview ASS upload and the join-audio upload.
**kn-data-sync** (`infrastructure/modules/kn_data_sync.py`):
- Memory goes from 512M to 1Gi. The 10-22 full sync OOMed at 489 MiB on
a retry. 1Gi also gives gen2 more CPU, and the cost is about $0 within
the free tier.
- `attempt_deadline="320s"` on both schedulers. The default 180s gave up
on a run that took 184.8s and succeeded, then retried it, which meant 3
full KN pulls that day. (1562752965629190217, 1562752969395535933)
## Testing
- `make test`: 6603 passed. 8 failed in `test_emulator_integration.py`,
the known local-only failure (another session's stack holds :8080); they
fail the same way on unmodified main, and CI's emulator job is
authoritative.
- Targeted runs: error monitor, audio-edit routes, Stripe/webhook,
conversion emails and the blocking guard all pass (710 tests). Frontend
crash-reporter jest: 17 passed, with new translated / not-translated
cases.
- The local CodeRabbit CLI failed with "Invalid token", so the bot
review is left on for this PR.
## Deploy note
The `infrastructure/` change (kn-data-sync) needs `pulumi up` before
merge, per the repo rule. That needs Andrew's `/gcp-admin` grant.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_015AvGQEptp3Pg9RHhKGZ5KB
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved responsiveness during audio editing and review submissions,
reducing the chance that file and audio processing delays other
requests.
* Reduced false crash reports on pages affected by machine translation.
* Improved recognition of startup probe failures in error monitoring.
* Clarified logging for malformed payment webhook signatures.
* **Reliability**
* Increased resources and allowed more time for scheduled data sync
jobs.
* Reduced delays to API requests during background processing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Alerts
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562375740685815953
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562224513276706933
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562541569238638725
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1561861805695967363
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562224514933334047
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562752965629190217
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562752969395535933
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
e4dad6e3c5
commit
f6d846234d
13 files changed
+146
-13
No files matched your search
@@ -36,6 +36,10 @@ RUN if [ -n "$MODEL_DIR" ]; then \
|
||||
pip install --no-cache-dir -e .; \
|
||||
fi
|
||||
|
||||
# Precompile bytecode so lazy imports never write .pyc at runtime: a cold
|
||||
# import that compiles + writes on the event loop has stalled requests 10-15s.
|
||||
RUN python -m compileall -q /app/backend /app/karaoke_gen
|
||||
|
||||
# Set environment variables
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
ENV PORT=8080
|
||||
|
||||
@@ -27,6 +27,7 @@ from backend.workers.video_worker import generate_video
|
||||
from backend.workers.render_video_worker import process_render_video
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.services.auth_service import AuthResult, UserType
|
||||
from backend.services.conversion_emails import process_conversion_emails
|
||||
from backend.services.job_manager import JobManager, PREP_PHASE_STATUSES
|
||||
from backend.utils.loop_bridge import run_in_body_thread, run_on_loop
|
||||
from backend.services.tracing import (
|
||||
@@ -668,7 +669,6 @@ def process_stale_reviews_endpoint(
|
||||
# Free-to-paid conversion emails ride on this hourly cron (no-op unless
|
||||
# CONVERSION_EMAILS_ENABLED); see backend/services/conversion_emails.py
|
||||
try:
|
||||
from backend.services.conversion_emails import process_conversion_emails
|
||||
conversion_result = await asyncio.to_thread(process_conversion_emails)
|
||||
logger.info(f"CONVERSION_EMAILS complete: {conversion_result}")
|
||||
except Exception as e:
|
||||
|
||||
@@ -2053,7 +2053,7 @@ async def generate_preview_video(
|
||||
# Upload ASS to GCS
|
||||
with create_span("upload-ass-to-gcs") as upload_ass_span:
|
||||
ass_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.ass"
|
||||
storage.upload_file(ass_path, ass_gcs_path)
|
||||
await asyncio.to_thread(storage.upload_file, ass_path, ass_gcs_path)
|
||||
upload_ass_span.set_attribute("ass_gcs_path", ass_gcs_path)
|
||||
|
||||
# A stale error marker from an earlier failed attempt at
|
||||
@@ -3074,13 +3074,14 @@ async def apply_audio_edit(
|
||||
if existing_stack:
|
||||
duration_before = existing_stack[-1].get('duration_seconds', 0)
|
||||
else:
|
||||
before_metadata = edit_service.get_metadata_from_gcs(current_gcs_path)
|
||||
before_metadata = await asyncio.to_thread(edit_service.get_metadata_from_gcs, current_gcs_path)
|
||||
duration_before = before_metadata.duration_seconds
|
||||
|
||||
output_gcs_path = f"jobs/{job_id}/audio_edit/edit_{edit_id}.flac"
|
||||
|
||||
# Apply the edit
|
||||
metadata, result_path = edit_service.apply_edit(
|
||||
# Apply the edit (ffmpeg + GCS download/upload: off the event loop)
|
||||
metadata, result_path = await asyncio.to_thread(
|
||||
edit_service.apply_edit,
|
||||
input_gcs_path=current_gcs_path,
|
||||
operation=operation,
|
||||
params=params,
|
||||
@@ -3285,7 +3286,7 @@ async def upload_audio_for_join(
|
||||
tmp_path = tmp.name
|
||||
|
||||
try:
|
||||
storage.upload_file(tmp_path, gcs_path)
|
||||
await asyncio.to_thread(storage.upload_file, tmp_path, gcs_path)
|
||||
|
||||
# Get metadata
|
||||
metadata = edit_service.get_metadata(tmp_path)
|
||||
@@ -3394,8 +3395,8 @@ async def submit_audio_edit(
|
||||
# Download and re-upload to canonical path
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
local_path = os.path.join(temp_dir, "edited.flac")
|
||||
storage.download_file(current_edited_path, local_path)
|
||||
storage.upload_file(local_path, edited_input_path)
|
||||
await asyncio.to_thread(storage.download_file, current_edited_path, local_path)
|
||||
await asyncio.to_thread(storage.upload_file, local_path, edited_input_path)
|
||||
|
||||
# Update job to use edited audio as input
|
||||
job_manager.update_job(job_id, {
|
||||
|
||||
@@ -38,7 +38,7 @@ class IgnoreReason:
|
||||
_IGNORE_PATTERNS: list[tuple[str, re.Pattern, str]] = [
|
||||
(
|
||||
"startup_probe",
|
||||
re.compile(r"startup probe failed", re.IGNORECASE),
|
||||
re.compile(r"startup (?:tcp |http |grpc )?probe failed", re.IGNORECASE),
|
||||
"Cloud Run cold start — startup probe failure is transient and expected",
|
||||
),
|
||||
(
|
||||
|
||||
@@ -19,6 +19,9 @@ from backend.services.fake_workers import fake_workers_enabled
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Stripe's SignatureVerificationError message when the header isn't "t=...,v1=..." at all.
|
||||
UNPARSEABLE_SIGNATURE_HEADER = "Unable to extract timestamp and signatures from header"
|
||||
|
||||
|
||||
# Credit packages available for purchase
|
||||
CREDIT_PACKAGES = {
|
||||
@@ -327,7 +330,13 @@ class StripeService:
|
||||
event_dict = json.loads(str(event))
|
||||
return True, event_dict, "Webhook verified"
|
||||
except stripe.error.SignatureVerificationError as e:
|
||||
logger.error(f"Invalid webhook signature: {e}")
|
||||
if UNPARSEABLE_SIGNATURE_HEADER in str(e):
|
||||
# Scanners/link-preview bots POST garbage headers; Stripe never sends one.
|
||||
logger.warning(f"Invalid webhook signature: {e}")
|
||||
else:
|
||||
# A well-formed header that doesn't match means a wrong webhook secret or a
|
||||
# tampered payload: every real delivery would fail, so keep it page-worthy.
|
||||
logger.error(f"Invalid webhook signature: {e}")
|
||||
return False, None, "Invalid signature"
|
||||
except Exception as e:
|
||||
logger.error(f"Error verifying webhook: {e}")
|
||||
|
||||
@@ -217,6 +217,12 @@ BLOCKING_CALLS = {
|
||||
"get_tenant_config", # GCS read on a cache miss (tenant middleware, up to 1.2s)
|
||||
"append_log_to_subcollection", # job-log write; on the loop it is offloaded, see log_write_offload
|
||||
"append_worker_log",
|
||||
# 2026-10-22 #alerts triage: audio-editor routes ran ffmpeg + GCS transfers on the loop
|
||||
# (storage_service.upload_file stalled 10.3s per edit).
|
||||
"apply_edit",
|
||||
"get_metadata_from_gcs",
|
||||
"upload_file",
|
||||
"download_file",
|
||||
}
|
||||
_BLOCKING_MODULE_CALLS = {("requests", "get"), ("requests", "post"), ("requests", "put"),
|
||||
("requests", "delete"), ("requests", "request"), ("time", "sleep")}
|
||||
|
||||
@@ -210,3 +210,30 @@ class TestStripeServiceMetadata:
|
||||
metadata = call_kwargs['metadata']
|
||||
|
||||
assert len(metadata['notes']) == 500
|
||||
|
||||
|
||||
class TestWebhookSignatureLogLevels:
|
||||
"""Scanner garbage logs WARNING; a well-formed but wrong signature stays ERROR (pages)."""
|
||||
|
||||
@pytest.fixture
|
||||
def stripe_service(self):
|
||||
with patch.dict(os.environ, {
|
||||
'STRIPE_SECRET_KEY': 'sk_test_fake',
|
||||
'STRIPE_WEBHOOK_SECRET': 'whsec_test_fake',
|
||||
}):
|
||||
from backend.services.stripe_service import StripeService
|
||||
return StripeService()
|
||||
|
||||
def test_unparseable_header_logs_warning(self, stripe_service, caplog):
|
||||
with caplog.at_level('WARNING', logger='backend.services.stripe_service'):
|
||||
ok, event, _ = stripe_service.verify_webhook_signature(b'{}', 'garbage')
|
||||
assert not ok and event is None
|
||||
levels = [r.levelname for r in caplog.records if 'Invalid webhook signature' in r.getMessage()]
|
||||
assert levels == ['WARNING']
|
||||
|
||||
def test_wrong_signature_logs_error(self, stripe_service, caplog):
|
||||
with caplog.at_level('WARNING', logger='backend.services.stripe_service'):
|
||||
ok, event, _ = stripe_service.verify_webhook_signature(b'{}', 't=1700000000,v1=deadbeef')
|
||||
assert not ok and event is None
|
||||
levels = [r.levelname for r in caplog.records if 'Invalid webhook signature' in r.getMessage()]
|
||||
assert levels == ['ERROR']
|
||||
@@ -6,6 +6,14 @@ Key insights for future AI agents working on this codebase.
|
||||
|
||||
---
|
||||
|
||||
## A lazy import on the loop can stall it for 15s: the image had no precompiled bytecode (Oct 2026, v0.282.1)
|
||||
|
||||
The hourly `process-stale-reviews` cron stalled the API loop 10–15s, 4 times in 2 days. The culprit was `internal.py:670`: a lazy `from backend.services.conversion_emails import ...` inside an `async def`. The stall stack ended in `importlib … _write_atomic`. The backend Dockerfile never ran `compileall`, so the first import on each instance compiled the module and wrote a `.pyc` on the event loop, on a CPU-throttled instance.
|
||||
|
||||
**Fix:** import at module top, and `RUN python -m compileall` over `/app/backend` and `/app/karaoke_gen` in `backend/Dockerfile`. decide had the same lesson for its cold start (decide #167).
|
||||
|
||||
**Also in this round:** the audio-editor routes ran ffmpeg and GCS transfers on the loop (10s per edit). `BLOCKING_CALLS` in `test_no_blocking_async_routes.py` now includes `apply_edit`, `get_metadata_from_gcs`, `upload_file` and `download_file`.
|
||||
|
||||
## A stateful fake backend finds bugs that static mocks can't (Oct 2026, v0.275.0)
|
||||
|
||||
The first run of the local-stack suite (real API on the emulators, fake workers) found that every job confirmed at "This song is longer than expected" would stall forever. `confirm-duration` moved the job to `separating_stage1`, which can only move on to `separating_stage2`, and the lyrics worker only advances jobs from `PREP_PHASE_STATUSES`. No unit test could see it: each one checked its own half (the route asserted it set `SEPARATING_STAGE1`, and the screens gate tests asserted the prep statuses). Only walking a job across the boundary did. No production job had confirmed a duration yet.
|
||||
|
||||
@@ -22,6 +22,44 @@ describe('isBenignError', () => {
|
||||
expect(isBenignError(abort)).toBe(true)
|
||||
})
|
||||
|
||||
describe('translate-induced DOM conflicts', () => {
|
||||
const removeChild = new Error(
|
||||
"Failed to execute 'removeChild' on 'Node': The node to be removed is not a child of this node."
|
||||
)
|
||||
removeChild.name = 'NotFoundError'
|
||||
const insertBefore = new Error(
|
||||
"Failed to execute 'insertBefore' on 'Node': The node before which the new node is to be inserted is not a child of this node."
|
||||
)
|
||||
insertBefore.name = 'NotFoundError'
|
||||
afterEach(() => {
|
||||
document.documentElement.className = ''
|
||||
})
|
||||
|
||||
it('is benign while the page is machine-translated', () => {
|
||||
document.documentElement.className = 'translated-ltr'
|
||||
expect(isBenignError(removeChild)).toBe(true)
|
||||
expect(isBenignError(insertBefore)).toBe(true)
|
||||
document.documentElement.className = 'dark translated-rtl'
|
||||
expect(isBenignError(removeChild)).toBe(true)
|
||||
})
|
||||
|
||||
it('is benign when translator markup is present without the <html> class', () => {
|
||||
const font = document.createElement('font')
|
||||
font.setAttribute('style', 'vertical-align: inherit;')
|
||||
document.body.appendChild(font)
|
||||
try {
|
||||
expect(isBenignError(removeChild)).toBe(true)
|
||||
} finally {
|
||||
font.remove()
|
||||
}
|
||||
})
|
||||
|
||||
it('is still reported when the page is not translated', () => {
|
||||
expect(isBenignError(removeChild)).toBe(false)
|
||||
expect(isBenignError(insertBefore)).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
it('does not suppress real errors', () => {
|
||||
expect(isBenignError(new Error('Cannot read properties of undefined'))).toBe(false)
|
||||
expect(
|
||||
|
||||
@@ -121,12 +121,30 @@ const BENIGN_MESSAGE_PATTERNS = [
|
||||
/resizeobserver loop/i,
|
||||
]
|
||||
|
||||
/**
|
||||
* Browser page translation (Chrome/Edge "Translate this page") rewrites the text
|
||||
* nodes React owns, so React's next commit throws NotFoundError from removeChild /
|
||||
* insertBefore. The crash is caused by the translator, not our code, and only
|
||||
* counts as benign while the page is actually translated (Chrome marks <html>
|
||||
* with a `translated-ltr` / `translated-rtl` class).
|
||||
*/
|
||||
const TRANSLATE_DOM_CONFLICT = /Failed to execute '(removeChild|insertBefore)' on 'Node'/
|
||||
|
||||
function isPageMachineTranslated(): boolean {
|
||||
if (typeof document === 'undefined') return false
|
||||
if (/\btranslated-(ltr|rtl)\b/.test(document.documentElement.className)) return true
|
||||
// Translators that don't mark <html> (e.g. Edge) still wrap the text they rewrite in
|
||||
// <font style="vertical-align: inherit">, which our app never renders.
|
||||
return document.querySelector('font[style*="vertical-align: inherit"]') !== null
|
||||
}
|
||||
|
||||
export function isBenignError(e: unknown): boolean {
|
||||
if (e instanceof DOMException && e.name === 'AbortError') return true
|
||||
// Some browsers surface the same media abort as a plain Error.
|
||||
if (e instanceof Error && e.name === 'AbortError') return true
|
||||
const message =
|
||||
e instanceof Error ? e.message : typeof e === 'string' ? e : ''
|
||||
if (TRANSLATE_DOM_CONFLICT.test(message) && isPageMachineTranslated()) return true
|
||||
return BENIGN_MESSAGE_PATTERNS.some((re) => re.test(message))
|
||||
}
|
||||
|
||||
|
||||
@@ -160,8 +160,12 @@ def create_kn_data_sync_resources(all_secrets: dict) -> dict:
|
||||
),
|
||||
),
|
||||
service_config=cloudfunctionsv2.FunctionServiceConfigArgs(
|
||||
available_memory="512M",
|
||||
timeout_seconds=300, # 5 minutes
|
||||
# 512M OOMed on the full sync (2026-10-22). The CPU has to be set with it:
|
||||
# the function was pinned at 0.333 vCPU, which caps memory at 512Mi.
|
||||
# A full vCPU also keeps the run well under the scheduler's attempt deadline.
|
||||
available_memory="1Gi",
|
||||
available_cpu="1",
|
||||
timeout_seconds=540, # 9 minutes; the full sync took up to ~185s at 0.33 vCPU
|
||||
min_instance_count=0,
|
||||
max_instance_count=1,
|
||||
service_account_email=sa.email,
|
||||
@@ -211,6 +215,9 @@ def create_kn_data_sync_resources(all_secrets: dict) -> dict:
|
||||
region=REGION,
|
||||
schedule="30 4 * * *",
|
||||
time_zone="America/New_York",
|
||||
# Above the function's 540s timeout, so the scheduler never gives up on
|
||||
# (and retries) a sync that is still running; the 180s default did.
|
||||
attempt_deadline="600s",
|
||||
http_target=cloudscheduler.JobHttpTargetArgs(
|
||||
uri=function.url,
|
||||
http_method="POST",
|
||||
@@ -237,6 +244,9 @@ def create_kn_data_sync_resources(all_secrets: dict) -> dict:
|
||||
region=REGION,
|
||||
schedule="5 3 * * *",
|
||||
time_zone="America/New_York",
|
||||
# Above the function's 540s timeout, so the scheduler never gives up on
|
||||
# (and retries) a sync that is still running; the 180s default did.
|
||||
attempt_deadline="600s",
|
||||
http_target=cloudscheduler.JobHttpTargetArgs(
|
||||
uri=function.url,
|
||||
http_method="POST",
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[tool.poetry]
|
||||
name = "karaoke-gen"
|
||||
version = "0.282.0"
|
||||
version = "0.282.1"
|
||||
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
|
||||
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
|
||||
license = "MIT"
|
||||
|
||||
@@ -43,6 +43,18 @@ class TestStartupProbePattern:
|
||||
assert result is not None
|
||||
assert result.pattern_name == "startup_probe"
|
||||
|
||||
def test_default_startup_tcp_probe(self):
|
||||
"""Cloud Run's own wording names the probe type (seen 2026-10-20)."""
|
||||
from backend.services.error_monitor.known_issues import should_ignore
|
||||
|
||||
msg = (
|
||||
'Default STARTUP TCP probe failed 1 time consecutively for container '
|
||||
'"karaoke-backend-1" on port 8080. The instance was not started.'
|
||||
)
|
||||
result = should_ignore("karaoke-backend", msg)
|
||||
assert result is not None
|
||||
assert result.pattern_name == "startup_probe"
|
||||
|
||||
def test_startup_probe_uppercase(self):
|
||||
from backend.services.error_monitor.known_issues import should_ignore
|
||||
|
||||
|
||||
Reference in new issue
Block a user