Files
Andrew Beveridge ☄️andClaude Opus 5.5 f6d846234d fix: alert noise + loop stalls + kn-data-sync OOM from #alerts triage (v0.282.1) (#1210)
## Summary
Fixes for the open and noise groups from the 2026-10-22 #alerts triage.

**Noise that should not page:**
- `stripe_service.py`: "Invalid webhook signature" now logs WARNING. All
42 hits on 10-21 came from `facebookexternalhit` via Cloudflare with a
malformed header. Every real Stripe delivery since 10-14 got a 200.
(#alerts 1562375740685815953)
- `error_monitor/known_issues.py`: `startup_probe` also matches Cloud
Run's "Default STARTUP **TCP** probe failed" wording.
(1562224513276706933)
- `frontend/lib/crash-reporter.ts`: a `removeChild`/`insertBefore`
NotFoundError is benign **only while the page is machine-translated**,
i.e. `<html>` has `translated-ltr`/`translated-rtl`. In the 10-21 crash,
Chrome had translated `/en/app` into Spanish ("Crea un video karaoke" is
not our string), and the customer's job completed and was downloaded.
(1562541569238638725)

**Event-loop stalls** (continues the WIP freeze item in BACKLOG):
- `review.py` audio-editor routes: `apply_edit`, `get_metadata_from_gcs`
and the GCS `download_file`/`upload_file` calls now go through
`asyncio.to_thread`. They were stalling the loop about 10s per edit (job
330d796c, 10-19). (1561861805695967363)
- `internal.py`: `process_conversion_emails` is imported at module top.
The lazy import inside the hourly cron's `async def` wrote a `.pyc` on
the loop, stalling it up to 15.3s, 4× on 10-20/21. (1562224514933334047)
- `backend/Dockerfile`: `python -m compileall` over `/app/backend` and
`/app/karaoke_gen`, so no lazy import compiles at runtime.
- `test_no_blocking_async_routes.py`: `BLOCKING_CALLS` now includes
`apply_edit`, `get_metadata_from_gcs`, `upload_file` and
`download_file`. Two more routes in `review.py` were caught and wrapped:
the preview ASS upload and the join-audio upload.

**kn-data-sync** (`infrastructure/modules/kn_data_sync.py`):
- Memory goes from 512M to 1Gi. The 10-22 full sync OOMed at 489 MiB on
a retry. 1Gi also gives gen2 more CPU, and the cost is about $0 within
the free tier.
- `attempt_deadline="320s"` on both schedulers. The default 180s gave up
on a run that took 184.8s and succeeded, then retried it, which meant 3
full KN pulls that day. (1562752965629190217, 1562752969395535933)

## Testing
- `make test`: 6603 passed. 8 failed in `test_emulator_integration.py`,
the known local-only failure (another session's stack holds :8080); they
fail the same way on unmodified main, and CI's emulator job is
authoritative.
- Targeted runs: error monitor, audio-edit routes, Stripe/webhook,
conversion emails and the blocking guard all pass (710 tests). Frontend
crash-reporter jest: 17 passed, with new translated / not-translated
cases.
- The local CodeRabbit CLI failed with "Invalid token", so the bot
review is left on for this PR.

## Deploy note
The `infrastructure/` change (kn-data-sync) needs `pulumi up` before
merge, per the repo rule. That needs Andrew's `/gcp-admin` grant.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_015AvGQEptp3Pg9RHhKGZ5KB

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved responsiveness during audio editing and review submissions,
reducing the chance that file and audio processing delays other
requests.
* Reduced false crash reports on pages affected by machine translation.
  * Improved recognition of startup probe failures in error monitoring.
  * Clarified logging for malformed payment webhook signatures.
* **Reliability**
* Increased resources and allowed more time for scheduled data sync
jobs.
  * Reduced delays to API requests during background processing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

## Alerts
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562375740685815953
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562224513276706933
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562541569238638725
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1561861805695967363
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562224514933334047
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562752965629190217
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562752969395535933

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 19:26:43 +00:00

269 lines
10 KiB
Python

"""
KaraokeNerds Data Sync infrastructure module.
Creates:
- GCS bucket for raw JSON exports
- Service account with BigQuery + GCS + Secret Manager permissions
- Cloud Function v2 for fetching KN data
- Two Cloud Scheduler jobs (full catalog + community tracks)
- BigQuery table for community tracks
"""
import pulumi
import pulumi_gcp as gcp
from pulumi_gcp import (
cloudfunctionsv2,
cloudscheduler,
serviceaccount,
storage,
bigquery,
)
from config import PROJECT_ID, REGION, get_project_number
BIGQUERY_DATASET = "karaoke_decide"
def create_kn_data_sync_resources(all_secrets: dict) -> dict:
"""Create all KN data sync infrastructure resources."""
resources = {}
# ==================== Service Account ====================
sa = serviceaccount.Account(
"kn-data-sync-sa",
account_id="kn-data-sync",
display_name="KaraokeNerds Data Sync Function",
description="Service account for the KN data sync Cloud Function",
)
resources["service_account"] = sa
# BigQuery Data Editor
resources["bq_access"] = gcp.projects.IAMMember(
"kn-data-sync-bigquery-access",
project=PROJECT_ID,
role="roles/bigquery.dataEditor",
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
# BigQuery Job User
resources["bq_job_user"] = gcp.projects.IAMMember(
"kn-data-sync-bigquery-job-user",
project=PROJECT_ID,
role="roles/bigquery.jobUser",
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Secret Manager Accessor (for KN API key)
resources["secrets_access"] = gcp.projects.IAMMember(
"kn-data-sync-secrets-access",
project=PROJECT_ID,
role="roles/secretmanager.secretAccessor",
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Logging Writer
resources["logging_access"] = gcp.projects.IAMMember(
"kn-data-sync-logging-access",
project=PROJECT_ID,
role="roles/logging.logWriter",
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
# ==================== GCS Bucket for exports ====================
data_bucket = storage.Bucket(
"kn-data-bucket",
name="nomadkaraoke-kn-data",
location="US-CENTRAL1",
force_destroy=False, # Keep historical exports
uniform_bucket_level_access=True,
lifecycle_rules=[
storage.BucketLifecycleRuleArgs(
action=storage.BucketLifecycleRuleActionArgs(type="Delete"),
condition=storage.BucketLifecycleRuleConditionArgs(
age=365, # Keep 1 year of exports
matches_prefixes=["full/full-data-2", "community/community-data-2"],
),
),
],
)
resources["data_bucket"] = data_bucket
# Grant SA write access to the data bucket
resources["bucket_access"] = storage.BucketIAMMember(
"kn-data-sync-bucket-access",
bucket=data_bucket.name,
role="roles/storage.objectAdmin",
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
# The KJ box (NomadPC) mirrors the KN exports into a local SQLite catalog
# so live-show searches don't depend on venue Wi-Fi / BigQuery. Its
# existing read-only sync identity (nomad-master-sync, the same SA that
# rsyncs NOMAD-720p masters from the divebar files bucket) gets read
# access here; the bucket itself stays private.
resources["kjbox_read_access"] = storage.BucketIAMMember(
"kn-data-kjbox-read-access",
bucket=data_bucket.name,
role="roles/storage.objectViewer",
member="serviceAccount:nomad-master-sync@nomadkaraoke.iam.gserviceaccount.com",
)
# ==================== Function Source Bucket ====================
source_bucket = storage.Bucket(
"kn-data-sync-function-source",
name=f"kn-data-sync-source-{PROJECT_ID}",
location="US-CENTRAL1",
force_destroy=True,
uniform_bucket_level_access=True,
)
resources["source_bucket"] = source_bucket
# ==================== BigQuery: Community Table ====================
# The karaokenerds_raw table already exists (managed by karaoke-decide).
# We add a new community table for the community tracks with YouTube URLs.
community_table = bigquery.Table(
"karaokenerds-community-table",
dataset_id=BIGQUERY_DATASET,
table_id="karaokenerds_community",
project=PROJECT_ID,
schema="""[
{"name": "Artist", "type": "STRING", "mode": "REQUIRED"},
{"name": "Title", "type": "STRING", "mode": "REQUIRED"},
{"name": "Brand", "type": "STRING", "mode": "REQUIRED"},
{"name": "Watch", "type": "STRING", "mode": "NULLABLE"},
{"name": "Created", "type": "STRING", "mode": "NULLABLE"},
{"name": "Id", "type": "INTEGER", "mode": "NULLABLE"}
]""",
deletion_protection=False,
)
resources["community_table"] = community_table
# ==================== Cloud Function ====================
function = cloudfunctionsv2.Function(
"kn-data-sync-function",
name="kn-data-sync",
location=REGION,
description="Syncs KaraokeNerds catalog data to BigQuery and GCS",
build_config=cloudfunctionsv2.FunctionBuildConfigArgs(
runtime="python312",
entry_point="sync_kn_data",
source=cloudfunctionsv2.FunctionBuildConfigSourceArgs(
storage_source=cloudfunctionsv2.FunctionBuildConfigSourceStorageSourceArgs(
bucket=source_bucket.name,
object="kn-data-sync-source.zip",
),
),
),
service_config=cloudfunctionsv2.FunctionServiceConfigArgs(
# 512M OOMed on the full sync (2026-10-22). The CPU has to be set with it:
# the function was pinned at 0.333 vCPU, which caps memory at 512Mi.
# A full vCPU also keeps the run well under the scheduler's attempt deadline.
available_memory="1Gi",
available_cpu="1",
timeout_seconds=540, # 9 minutes; the full sync took up to ~185s at 0.33 vCPU
min_instance_count=0,
max_instance_count=1,
service_account_email=sa.email,
environment_variables={
"GCP_PROJECT_ID": PROJECT_ID,
"GCS_BUCKET": "nomadkaraoke-kn-data",
},
secret_environment_variables=[
cloudfunctionsv2.FunctionServiceConfigSecretEnvironmentVariableArgs(
key="KARAOKENERDS_API_KEY",
project_id=PROJECT_ID,
secret=all_secrets["karaokenerds-api-key"].secret_id,
version="latest",
),
],
),
)
resources["function"] = function
# ==================== IAM: Cloud Scheduler → Function ====================
resources["sa_run_invoker"] = gcp.cloudrunv2.ServiceIamMember(
"kn-data-sync-sa-run-invoker",
project=PROJECT_ID,
location=REGION,
name=function.name,
role="roles/run.invoker",
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
resources["sa_cf_invoker"] = cloudfunctionsv2.FunctionIamMember(
"kn-data-sync-sa-cf-invoker",
project=PROJECT_ID,
location=REGION,
cloud_function=function.name,
role="roles/cloudfunctions.invoker",
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
# ==================== Cloud Scheduler Jobs ====================
# Full catalog sync — daily at 4:30 AM ET
resources["full_scheduler"] = cloudscheduler.Job(
"kn-data-sync-full-scheduler",
name="kn-data-sync-full-daily",
description="Daily sync of full KaraokeNerds song catalog to BigQuery",
region=REGION,
schedule="30 4 * * *",
time_zone="America/New_York",
# Above the function's 540s timeout, so the scheduler never gives up on
# (and retries) a sync that is still running; the 180s default did.
attempt_deadline="600s",
http_target=cloudscheduler.JobHttpTargetArgs(
uri=function.url,
http_method="POST",
body=pulumi.Output.from_input(
'{"mode": "full"}'
).apply(lambda b: __import__("base64").b64encode(b.encode()).decode()),
headers={"Content-Type": "application/json"},
oidc_token=cloudscheduler.JobHttpTargetOidcTokenArgs(
service_account_email=sa.email,
),
),
retry_config=cloudscheduler.JobRetryConfigArgs(
retry_count=2,
min_backoff_duration="60s",
max_backoff_duration="300s",
),
)
# Community sync — daily at 3:05 AM ET
resources["community_scheduler"] = cloudscheduler.Job(
"kn-data-sync-community-scheduler",
name="kn-data-sync-community-daily",
description="Daily sync of KaraokeNerds community tracks to BigQuery",
region=REGION,
schedule="5 3 * * *",
time_zone="America/New_York",
# Above the function's 540s timeout, so the scheduler never gives up on
# (and retries) a sync that is still running; the 180s default did.
attempt_deadline="600s",
http_target=cloudscheduler.JobHttpTargetArgs(
uri=function.url,
http_method="POST",
body=pulumi.Output.from_input(
'{"mode": "community"}'
).apply(lambda b: __import__("base64").b64encode(b.encode()).decode()),
headers={"Content-Type": "application/json"},
oidc_token=cloudscheduler.JobHttpTargetOidcTokenArgs(
service_account_email=sa.email,
),
),
retry_config=cloudscheduler.JobRetryConfigArgs(
retry_count=2,
min_backoff_duration="60s",
max_backoff_duration="300s",
),
)
return resources