mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
infra: GCP cost cuts (VPC connector, min-instances, Autoclass, weekly incremental git backup, hourly error monitor, PITR off) (#1063)
## Summary These are the cost cuts Andrew approved to get the whole GCP project under $300/mo. Performance compromises were accepted. - **VPC connector removed.** `cloud-run-connector` only existed to reach the old flacfetch GCE VM's private IP. flacfetch now runs on Netcup behind a public HTTPS URL. I checked every URL secret the backend uses and none are private IPs. The backend is detached with `--clear-vpc-connector`: just dropping the `--vpc-connector` flag from `gcloud run deploy` would keep the old setting. `audio-download-job` and `bulk-search-job` no longer set `vpc_access`. Saves ~$12/mo plus the Network Intelligence resource-hours. - **`karaoke-backend` min-instances 2 → 1** (ci.yml). - **Autoclass** on `karaoke-gen-storage-nomadkaraoke` (terminal NEARLINE, because customers re-download old jobs) and on `nomadkaraoke-divebar-files` (terminal ARCHIVE). Autoclass charges no retrieval or early-deletion fees. - **backup-to-aws** - **Git bundles:** now weekly (Sundays) and incremental. A repo is re-bundled only when its GitHub `pushed_at` is newer than its bundle's S3 `LastModified` minus a 1h lag, so S3 itself is the state store. Every repo is re-bundled on the first Sunday of each month. `?mode=git_repos[&full=1]` runs the git step alone. - **Bug fix:** since #997 every S3 **overwrite** has failed. The `backup-writer` IAM user has no `s3:GetObject`, so HEAD on an existing key returns 403. Git bundles and the manifest were stuck at 09-15, and the DR freshness monitor has been red every day. The existence check now uses `ListObjectsV2`. - **`gcs_sync`:** detects changes by `time_created` instead of `updated`. Autoclass transitions bump `updated`, which would have made the sync re-copy every object and pull it back to Standard. - **DR freshness monitor:** `git-repos/` is now judged on `manifest.json` (192h) plus a check of the manifest's contents. - **Error monitor:** runs hourly instead of every 15 min. Lookback is 60 min, equal to the interval, so there are no gaps or double counts. `MAX_LOG_ENTRIES` goes from 500 to 2000. - **Firestore PITR disabled.** The nightly export to GCS staging remains, plus the weekly copy to S3. - **`recover-stuck-downloads` / `retry-pending-render-jobs`:** every 5 → 10 min. Both are recovery sweeps with ~10-min stall thresholds. - Docs updated: DISASTER-RECOVERY (PITR restore replaced, git RPO, incident write-up), ARCHITECTURE, LESSONS-LEARNED, GCP-COST-OPTIMIZATION, TROUBLESHOOTING. **Kept:** the `audio-separator` GPU service. Its request logs show ~40 external `/separate` calls/month from remote-CLI users. It's already min-instances 0. ## Already applied to prod (before merge) - `gcloud run services update karaoke-backend --clear-vpc-connector --min-instances 1`, same image. Health is OK, and `/api/health/flacfetch` and `/api/health/encoding-worker` are both OK without the connector. - Targeted `pulumi up` on the Firestore DB, both buckets, both jobs and both 5-min schedulers. - backup-to-aws redeployed with `deploy.sh`. A manual `?mode=git_repos` run bundled 13 of 121 repos (106 unchanged, 2 skipped) and uploaded 120 files with 0 errors. - Deleted `gs://…/backups/flacfetch-torrent-backup-20260122-1931.tar.gz` (1 GB). ## After merge Targeted `pulumi up` for `error-monitor-scheduler`, which must wait for the image with LOOKBACK=60, then delete the connector and drop the `vpcaccess` API service from state (`disable_on_destroy=False`, so the API itself stays enabled). ## Tests `make test` passes locally. The backup_to_aws function tests (43, including new incremental, list-based-existence and gcs_sync tests) were run separately because CI doesn't run them. Related: nomadkaraoke/karaoke-decide#140 (ARCHIVE lifecycle on `nomadkaraoke-data`, already applied). 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Updates** * Pending-render retries and stuck-download recovery now run every 10 minutes instead of every 5 minutes. * Error monitoring now runs hourly and checks a 60-minute window. * Cloud Run candidate deployments use one minimum instance instead of two and no longer use a VPC connector. * **Backups and Recovery** * Git repository backups now run weekly, with monthly full refreshes and incremental updates between them. * Firestore point-in-time recovery is disabled; recovery uses nightly exports. * Storage buckets now use automatic storage-class management. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
7179e2863f
commit
8cbcc03099
34 files changed
+780
-232
No files matched your search
@@ -2032,9 +2032,11 @@ jobs:
|
||||
# that but overshot once the loop was made non-blocking: every burst
|
||||
# forced scale-out, and each overflow request waited out a ~16s app
|
||||
# cold start (2026-09-22 16-tab test: warm instance served in 1.2s,
|
||||
# freshly-spawned ones in 16-19s). 32 + --min-instances 2 gives 64
|
||||
# warm slots so a 16-tab reload is absorbed without cold starts,
|
||||
# while staying far below the wedge-era 80.
|
||||
# freshly-spawned ones in 16-19s). 32 + --min-instances 2 gave 64
|
||||
# warm slots; min-instances was cut to 1 on 2026-09-26 (GCP cost cut,
|
||||
# accepted perf trade-off: big bursts may hit a ~15s cold start) while
|
||||
# staying far below the wedge-era 80. --clear-vpc-connector: the
|
||||
# serverless VPC connector was deleted (flacfetch is public now).
|
||||
gcloud run deploy karaoke-backend \
|
||||
--image $IMAGE \
|
||||
--no-traffic \
|
||||
@@ -2050,9 +2052,8 @@ jobs:
|
||||
--timeout 1800 \
|
||||
--concurrency 32 \
|
||||
--max-instances 20 \
|
||||
--min-instances 2 \
|
||||
--vpc-connector projects/nomadkaraoke/locations/us-central1/connectors/cloud-run-connector \
|
||||
--vpc-egress private-ranges-only \
|
||||
--min-instances 1 \
|
||||
--clear-vpc-connector \
|
||||
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,DEFAULT_PRIVATE_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-NonPublished,DEFAULT_PRIVATE_BRAND_PREFIX=NOMADNP,USE_GCE_ENCODING=true,USE_GCE_PREVIEW_ENCODING=true,ENABLE_PUSH_NOTIFICATIONS=true,USE_CLOUD_RUN_JOBS_FOR_VIDEO=true,USE_CLOUD_RUN_JOBS_FOR_RENDER=true,EMAIL_PROVIDER=postmark,AUTO_CORRECT_MODEL=claude-opus-5-5,AUTO_CORRECT_COMPARE_MODELS=claude-opus-5-5;gemini-3.8-flash,AUTO_CORRECT_PROACTIVE_ENABLED=true,AUTO_APPROVAL_BACKING_KEEP_ENABLED=true,COMMUNITY_DAILY_PICK_ENABLED=true,EDGE_AUTH_MODE=enforce,COMMIT_SHA=${{ steps.build-meta.outputs.sha }},PR_NUMBER=${{ steps.build-meta.outputs.pr_number }},PR_TITLE=${{ steps.build-meta.outputs.pr_title }}" \
|
||||
--set-secrets "ANTHROPIC_API_KEY=anthropic-api-key:latest,AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest,POSTMARK_SERVER_TOKEN=postmark-server-token:latest,STRIPE_SECRET_KEY=stripe-secret-key:latest,STRIPE_WEBHOOK_SECRET=stripe-webhook-secret:latest,LANGFUSE_PUBLIC_KEY=langfuse-public-key:latest,LANGFUSE_SECRET_KEY=langfuse-secret-key:latest,LANGFUSE_HOST=langfuse-host:latest,ENCODING_WORKER_URL=encoding-worker-url:latest,ENCODING_WORKER_API_KEY=encoding-worker-api-key:latest,ENCODING_WORKER_FALLBACK_VMS=encoding-worker-fallback-vms:latest,VAPID_PUBLIC_KEY=vapid-public-key:latest,VAPID_PRIVATE_KEY=vapid-private-key:latest,E2E_BYPASS_KEY=e2e-bypass-key:latest,GDRIVE_VALIDATOR_URL=gdrive-validator-url:latest,EDGE_ORIGIN_SECRET=edge-origin-secret:latest,KARAOKEHUNT_FORWARDER_SECRET=karaokehunt-forwarder-secret:latest,KJBOX_PARTNER_SECRET=kjbox-partner-secret:latest"
|
||||
|
||||
|
||||
@@ -55,21 +55,25 @@ jobs:
|
||||
BUCKET = os.environ["BUCKET"]
|
||||
# (label, prefix, max_age_hours, require_suffix)
|
||||
# require_suffix: only objects with this key suffix count toward
|
||||
# freshness (None = any object). git-repos/ requires ".bundle" so the
|
||||
# always-written manifest.json can't mark the prefix fresh when every
|
||||
# bundle failed / nothing was bundled — 0 bundles then reads as stale.
|
||||
# freshness (None = any object).
|
||||
# NOTE: Firestore and BigQuery export to GCS staging nightly but are
|
||||
# only uploaded to S3 *weekly* (Sundays) to cut cross-cloud egress —
|
||||
# see backup_to_aws/main.py (firestore_to_s3_today). This monitor only
|
||||
# sees the S3 (off-site) copy, so their limits must allow a full week
|
||||
# plus slack (192h ≈ 8 days), NOT the 36h nightly cadence. GCS job
|
||||
# files and secrets DO upload to S3 nightly, so they stay at 36h.
|
||||
# Git repos are also weekly (Sundays) AND incremental since
|
||||
# 2026-09-26 — only repos pushed that week get a fresh bundle, so a
|
||||
# quiet week legitimately leaves every .bundle older than 8 days.
|
||||
# Freshness is judged on git-repos/manifest.json instead (rewritten
|
||||
# every weekly run), and the manifest's contents are checked below so
|
||||
# a run where every repo failed still reads as stale.
|
||||
TARGETS = [
|
||||
("Firestore", "firestore/", 192, None),
|
||||
("GCS job files", "gcs/job-files/", 36, None),
|
||||
("Encrypted secrets", "secrets/", 36, None),
|
||||
("BigQuery weekly", "bigquery/daily-refresh/", 192, None),
|
||||
("Git repos", "git-repos/", 36, ".bundle"),
|
||||
("Git repos", "git-repos/manifest.json", 192, None),
|
||||
]
|
||||
|
||||
s3 = boto3.client("s3")
|
||||
@@ -97,6 +101,19 @@ jobs:
|
||||
line = f"{label}: {age_h}h old (limit {max_hours}h) — last {latest.isoformat()}"
|
||||
(stale if age > max_hours * 3600 else ok).append(line)
|
||||
|
||||
# Git manifest contents: ANY bundle error alerts, even though the
|
||||
# manifest itself is fresh. The git step is weekly + incremental, so
|
||||
# a repo that failed keeps an ever-older S3 bundle until the next
|
||||
# Sunday — silence here would let it drift past the 7-day RPO.
|
||||
try:
|
||||
import json as _json
|
||||
m = _json.loads(s3.get_object(Bucket=BUCKET, Key="git-repos/manifest.json")["Body"].read())
|
||||
if int(m.get("errors", 0)) > 0:
|
||||
failed = [r.get("full_name") for r in m.get("repos", []) if r.get("status") == "error"]
|
||||
stale.append(f"Git repos: last run had {m.get('errors')} bundle errors: {', '.join(failed[:10])}")
|
||||
except Exception as e: # noqa: BLE001
|
||||
stale.append(f"Git repos: could not read manifest.json ({e})")
|
||||
|
||||
summary = ["## DR backup freshness", ""]
|
||||
if stale:
|
||||
summary.append("### Stale")
|
||||
|
||||
Reference in new issue
Block a user