mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary
Fixes for the open and noise groups from the 2026-10-22 #alerts triage.
**Noise that should not page:**
- `stripe_service.py`: "Invalid webhook signature" now logs WARNING. All
42 hits on 10-21 came from `facebookexternalhit` via Cloudflare with a
malformed header. Every real Stripe delivery since 10-14 got a 200.
(#alerts 1562375740685815953)
- `error_monitor/known_issues.py`: `startup_probe` also matches Cloud
Run's "Default STARTUP **TCP** probe failed" wording.
(1562224513276706933)
- `frontend/lib/crash-reporter.ts`: a `removeChild`/`insertBefore`
NotFoundError is benign **only while the page is machine-translated**,
i.e. `<html>` has `translated-ltr`/`translated-rtl`. In the 10-21 crash,
Chrome had translated `/en/app` into Spanish ("Crea un video karaoke" is
not our string), and the customer's job completed and was downloaded.
(1562541569238638725)
**Event-loop stalls** (continues the WIP freeze item in BACKLOG):
- `review.py` audio-editor routes: `apply_edit`, `get_metadata_from_gcs`
and the GCS `download_file`/`upload_file` calls now go through
`asyncio.to_thread`. They were stalling the loop about 10s per edit (job
330d796c, 10-19). (1561861805695967363)
- `internal.py`: `process_conversion_emails` is imported at module top.
The lazy import inside the hourly cron's `async def` wrote a `.pyc` on
the loop, stalling it up to 15.3s, 4× on 10-20/21. (1562224514933334047)
- `backend/Dockerfile`: `python -m compileall` over `/app/backend` and
`/app/karaoke_gen`, so no lazy import compiles at runtime.
- `test_no_blocking_async_routes.py`: `BLOCKING_CALLS` now includes
`apply_edit`, `get_metadata_from_gcs`, `upload_file` and
`download_file`. Two more routes in `review.py` were caught and wrapped:
the preview ASS upload and the join-audio upload.
**kn-data-sync** (`infrastructure/modules/kn_data_sync.py`):
- Memory goes from 512M to 1Gi. The 10-22 full sync OOMed at 489 MiB on
a retry. 1Gi also gives gen2 more CPU, and the cost is about $0 within
the free tier.
- `attempt_deadline="320s"` on both schedulers. The default 180s gave up
on a run that took 184.8s and succeeded, then retried it, which meant 3
full KN pulls that day. (1562752965629190217, 1562752969395535933)
## Testing
- `make test`: 6603 passed. 8 failed in `test_emulator_integration.py`,
the known local-only failure (another session's stack holds :8080); they
fail the same way on unmodified main, and CI's emulator job is
authoritative.
- Targeted runs: error monitor, audio-edit routes, Stripe/webhook,
conversion emails and the blocking guard all pass (710 tests). Frontend
crash-reporter jest: 17 passed, with new translated / not-translated
cases.
- The local CodeRabbit CLI failed with "Invalid token", so the bot
review is left on for this PR.
## Deploy note
The `infrastructure/` change (kn-data-sync) needs `pulumi up` before
merge, per the repo rule. That needs Andrew's `/gcp-admin` grant.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_015AvGQEptp3Pg9RHhKGZ5KB
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved responsiveness during audio editing and review submissions,
reducing the chance that file and audio processing delays other
requests.
* Reduced false crash reports on pages affected by machine translation.
* Improved recognition of startup probe failures in error monitoring.
* Clarified logging for malformed payment webhook signatures.
* **Reliability**
* Increased resources and allowed more time for scheduled data sync
jobs.
* Reduced delays to API requests during background processing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Alerts
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562375740685815953
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562224513276706933
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562541569238638725
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1561861805695967363
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562224514933334047
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562752965629190217
Alert:
https://discord.com/channels/1198691101943541770/1265517528608669707/1562752969395535933
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
269 lines
10 KiB
Python
269 lines
10 KiB
Python
"""
|
|
KaraokeNerds Data Sync infrastructure module.
|
|
|
|
Creates:
|
|
- GCS bucket for raw JSON exports
|
|
- Service account with BigQuery + GCS + Secret Manager permissions
|
|
- Cloud Function v2 for fetching KN data
|
|
- Two Cloud Scheduler jobs (full catalog + community tracks)
|
|
- BigQuery table for community tracks
|
|
"""
|
|
|
|
import pulumi
|
|
import pulumi_gcp as gcp
|
|
from pulumi_gcp import (
|
|
cloudfunctionsv2,
|
|
cloudscheduler,
|
|
serviceaccount,
|
|
storage,
|
|
bigquery,
|
|
)
|
|
|
|
from config import PROJECT_ID, REGION, get_project_number
|
|
|
|
BIGQUERY_DATASET = "karaoke_decide"
|
|
|
|
|
|
def create_kn_data_sync_resources(all_secrets: dict) -> dict:
|
|
"""Create all KN data sync infrastructure resources."""
|
|
resources = {}
|
|
|
|
# ==================== Service Account ====================
|
|
|
|
sa = serviceaccount.Account(
|
|
"kn-data-sync-sa",
|
|
account_id="kn-data-sync",
|
|
display_name="KaraokeNerds Data Sync Function",
|
|
description="Service account for the KN data sync Cloud Function",
|
|
)
|
|
resources["service_account"] = sa
|
|
|
|
# BigQuery Data Editor
|
|
resources["bq_access"] = gcp.projects.IAMMember(
|
|
"kn-data-sync-bigquery-access",
|
|
project=PROJECT_ID,
|
|
role="roles/bigquery.dataEditor",
|
|
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
|
|
)
|
|
|
|
# BigQuery Job User
|
|
resources["bq_job_user"] = gcp.projects.IAMMember(
|
|
"kn-data-sync-bigquery-job-user",
|
|
project=PROJECT_ID,
|
|
role="roles/bigquery.jobUser",
|
|
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
|
|
)
|
|
|
|
# Secret Manager Accessor (for KN API key)
|
|
resources["secrets_access"] = gcp.projects.IAMMember(
|
|
"kn-data-sync-secrets-access",
|
|
project=PROJECT_ID,
|
|
role="roles/secretmanager.secretAccessor",
|
|
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
|
|
)
|
|
|
|
# Logging Writer
|
|
resources["logging_access"] = gcp.projects.IAMMember(
|
|
"kn-data-sync-logging-access",
|
|
project=PROJECT_ID,
|
|
role="roles/logging.logWriter",
|
|
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
|
|
)
|
|
|
|
# ==================== GCS Bucket for exports ====================
|
|
|
|
data_bucket = storage.Bucket(
|
|
"kn-data-bucket",
|
|
name="nomadkaraoke-kn-data",
|
|
location="US-CENTRAL1",
|
|
force_destroy=False, # Keep historical exports
|
|
uniform_bucket_level_access=True,
|
|
lifecycle_rules=[
|
|
storage.BucketLifecycleRuleArgs(
|
|
action=storage.BucketLifecycleRuleActionArgs(type="Delete"),
|
|
condition=storage.BucketLifecycleRuleConditionArgs(
|
|
age=365, # Keep 1 year of exports
|
|
matches_prefixes=["full/full-data-2", "community/community-data-2"],
|
|
),
|
|
),
|
|
],
|
|
)
|
|
resources["data_bucket"] = data_bucket
|
|
|
|
# Grant SA write access to the data bucket
|
|
resources["bucket_access"] = storage.BucketIAMMember(
|
|
"kn-data-sync-bucket-access",
|
|
bucket=data_bucket.name,
|
|
role="roles/storage.objectAdmin",
|
|
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
|
|
)
|
|
|
|
# The KJ box (NomadPC) mirrors the KN exports into a local SQLite catalog
|
|
# so live-show searches don't depend on venue Wi-Fi / BigQuery. Its
|
|
# existing read-only sync identity (nomad-master-sync, the same SA that
|
|
# rsyncs NOMAD-720p masters from the divebar files bucket) gets read
|
|
# access here; the bucket itself stays private.
|
|
resources["kjbox_read_access"] = storage.BucketIAMMember(
|
|
"kn-data-kjbox-read-access",
|
|
bucket=data_bucket.name,
|
|
role="roles/storage.objectViewer",
|
|
member="serviceAccount:nomad-master-sync@nomadkaraoke.iam.gserviceaccount.com",
|
|
)
|
|
|
|
# ==================== Function Source Bucket ====================
|
|
|
|
source_bucket = storage.Bucket(
|
|
"kn-data-sync-function-source",
|
|
name=f"kn-data-sync-source-{PROJECT_ID}",
|
|
location="US-CENTRAL1",
|
|
force_destroy=True,
|
|
uniform_bucket_level_access=True,
|
|
)
|
|
resources["source_bucket"] = source_bucket
|
|
|
|
# ==================== BigQuery: Community Table ====================
|
|
|
|
# The karaokenerds_raw table already exists (managed by karaoke-decide).
|
|
# We add a new community table for the community tracks with YouTube URLs.
|
|
community_table = bigquery.Table(
|
|
"karaokenerds-community-table",
|
|
dataset_id=BIGQUERY_DATASET,
|
|
table_id="karaokenerds_community",
|
|
project=PROJECT_ID,
|
|
schema="""[
|
|
{"name": "Artist", "type": "STRING", "mode": "REQUIRED"},
|
|
{"name": "Title", "type": "STRING", "mode": "REQUIRED"},
|
|
{"name": "Brand", "type": "STRING", "mode": "REQUIRED"},
|
|
{"name": "Watch", "type": "STRING", "mode": "NULLABLE"},
|
|
{"name": "Created", "type": "STRING", "mode": "NULLABLE"},
|
|
{"name": "Id", "type": "INTEGER", "mode": "NULLABLE"}
|
|
]""",
|
|
deletion_protection=False,
|
|
)
|
|
resources["community_table"] = community_table
|
|
|
|
# ==================== Cloud Function ====================
|
|
|
|
function = cloudfunctionsv2.Function(
|
|
"kn-data-sync-function",
|
|
name="kn-data-sync",
|
|
location=REGION,
|
|
description="Syncs KaraokeNerds catalog data to BigQuery and GCS",
|
|
build_config=cloudfunctionsv2.FunctionBuildConfigArgs(
|
|
runtime="python312",
|
|
entry_point="sync_kn_data",
|
|
source=cloudfunctionsv2.FunctionBuildConfigSourceArgs(
|
|
storage_source=cloudfunctionsv2.FunctionBuildConfigSourceStorageSourceArgs(
|
|
bucket=source_bucket.name,
|
|
object="kn-data-sync-source.zip",
|
|
),
|
|
),
|
|
),
|
|
service_config=cloudfunctionsv2.FunctionServiceConfigArgs(
|
|
# 512M OOMed on the full sync (2026-10-22). The CPU has to be set with it:
|
|
# the function was pinned at 0.333 vCPU, which caps memory at 512Mi.
|
|
# A full vCPU also keeps the run well under the scheduler's attempt deadline.
|
|
available_memory="1Gi",
|
|
available_cpu="1",
|
|
timeout_seconds=540, # 9 minutes; the full sync took up to ~185s at 0.33 vCPU
|
|
min_instance_count=0,
|
|
max_instance_count=1,
|
|
service_account_email=sa.email,
|
|
environment_variables={
|
|
"GCP_PROJECT_ID": PROJECT_ID,
|
|
"GCS_BUCKET": "nomadkaraoke-kn-data",
|
|
},
|
|
secret_environment_variables=[
|
|
cloudfunctionsv2.FunctionServiceConfigSecretEnvironmentVariableArgs(
|
|
key="KARAOKENERDS_API_KEY",
|
|
project_id=PROJECT_ID,
|
|
secret=all_secrets["karaokenerds-api-key"].secret_id,
|
|
version="latest",
|
|
),
|
|
],
|
|
),
|
|
)
|
|
resources["function"] = function
|
|
|
|
# ==================== IAM: Cloud Scheduler → Function ====================
|
|
|
|
resources["sa_run_invoker"] = gcp.cloudrunv2.ServiceIamMember(
|
|
"kn-data-sync-sa-run-invoker",
|
|
project=PROJECT_ID,
|
|
location=REGION,
|
|
name=function.name,
|
|
role="roles/run.invoker",
|
|
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
|
|
)
|
|
|
|
resources["sa_cf_invoker"] = cloudfunctionsv2.FunctionIamMember(
|
|
"kn-data-sync-sa-cf-invoker",
|
|
project=PROJECT_ID,
|
|
location=REGION,
|
|
cloud_function=function.name,
|
|
role="roles/cloudfunctions.invoker",
|
|
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
|
|
)
|
|
|
|
# ==================== Cloud Scheduler Jobs ====================
|
|
|
|
# Full catalog sync — daily at 4:30 AM ET
|
|
resources["full_scheduler"] = cloudscheduler.Job(
|
|
"kn-data-sync-full-scheduler",
|
|
name="kn-data-sync-full-daily",
|
|
description="Daily sync of full KaraokeNerds song catalog to BigQuery",
|
|
region=REGION,
|
|
schedule="30 4 * * *",
|
|
time_zone="America/New_York",
|
|
# Above the function's 540s timeout, so the scheduler never gives up on
|
|
# (and retries) a sync that is still running; the 180s default did.
|
|
attempt_deadline="600s",
|
|
http_target=cloudscheduler.JobHttpTargetArgs(
|
|
uri=function.url,
|
|
http_method="POST",
|
|
body=pulumi.Output.from_input(
|
|
'{"mode": "full"}'
|
|
).apply(lambda b: __import__("base64").b64encode(b.encode()).decode()),
|
|
headers={"Content-Type": "application/json"},
|
|
oidc_token=cloudscheduler.JobHttpTargetOidcTokenArgs(
|
|
service_account_email=sa.email,
|
|
),
|
|
),
|
|
retry_config=cloudscheduler.JobRetryConfigArgs(
|
|
retry_count=2,
|
|
min_backoff_duration="60s",
|
|
max_backoff_duration="300s",
|
|
),
|
|
)
|
|
|
|
# Community sync — daily at 3:05 AM ET
|
|
resources["community_scheduler"] = cloudscheduler.Job(
|
|
"kn-data-sync-community-scheduler",
|
|
name="kn-data-sync-community-daily",
|
|
description="Daily sync of KaraokeNerds community tracks to BigQuery",
|
|
region=REGION,
|
|
schedule="5 3 * * *",
|
|
time_zone="America/New_York",
|
|
# Above the function's 540s timeout, so the scheduler never gives up on
|
|
# (and retries) a sync that is still running; the 180s default did.
|
|
attempt_deadline="600s",
|
|
http_target=cloudscheduler.JobHttpTargetArgs(
|
|
uri=function.url,
|
|
http_method="POST",
|
|
body=pulumi.Output.from_input(
|
|
'{"mode": "community"}'
|
|
).apply(lambda b: __import__("base64").b64encode(b.encode()).decode()),
|
|
headers={"Content-Type": "application/json"},
|
|
oidc_token=cloudscheduler.JobHttpTargetOidcTokenArgs(
|
|
service_account_email=sa.email,
|
|
),
|
|
),
|
|
retry_config=cloudscheduler.JobRetryConfigArgs(
|
|
retry_count=2,
|
|
min_backoff_duration="60s",
|
|
max_backoff_duration="300s",
|
|
),
|
|
)
|
|
|
|
return resources
|