infra: GCP cost cuts (VPC connector, min-instances, Autoclass, weekly incremental git backup, hourly error monitor, PITR off) (#1063)

## Summary
These are the cost cuts Andrew approved to get the whole GCP project
under $300/mo. Performance compromises were accepted.

- **VPC connector removed.** `cloud-run-connector` only existed to reach
the old flacfetch GCE VM's private IP. flacfetch now runs on Netcup
behind a public HTTPS URL. I checked every URL secret the backend uses
and none are private IPs. The backend is detached with
`--clear-vpc-connector`: just dropping the `--vpc-connector` flag from
`gcloud run deploy` would keep the old setting. `audio-download-job` and
`bulk-search-job` no longer set `vpc_access`. Saves ~$12/mo plus the
Network Intelligence resource-hours.
- **`karaoke-backend` min-instances 2 → 1** (ci.yml).
- **Autoclass** on `karaoke-gen-storage-nomadkaraoke` (terminal
NEARLINE, because customers re-download old jobs) and on
`nomadkaraoke-divebar-files` (terminal ARCHIVE). Autoclass charges no
retrieval or early-deletion fees.
- **backup-to-aws**
- **Git bundles:** now weekly (Sundays) and incremental. A repo is
re-bundled only when its GitHub `pushed_at` is newer than its bundle's
S3 `LastModified` minus a 1h lag, so S3 itself is the state store. Every
repo is re-bundled on the first Sunday of each month.
`?mode=git_repos[&full=1]` runs the git step alone.
- **Bug fix:** since #997 every S3 **overwrite** has failed. The
`backup-writer` IAM user has no `s3:GetObject`, so HEAD on an existing
key returns 403. Git bundles and the manifest were stuck at 09-15, and
the DR freshness monitor has been red every day. The existence check now
uses `ListObjectsV2`.
- **`gcs_sync`:** detects changes by `time_created` instead of
`updated`. Autoclass transitions bump `updated`, which would have made
the sync re-copy every object and pull it back to Standard.
- **DR freshness monitor:** `git-repos/` is now judged on
`manifest.json` (192h) plus a check of the manifest's contents.
- **Error monitor:** runs hourly instead of every 15 min. Lookback is 60
min, equal to the interval, so there are no gaps or double counts.
`MAX_LOG_ENTRIES` goes from 500 to 2000.
- **Firestore PITR disabled.** The nightly export to GCS staging
remains, plus the weekly copy to S3.
- **`recover-stuck-downloads` / `retry-pending-render-jobs`:** every 5 →
10 min. Both are recovery sweeps with ~10-min stall thresholds.
- Docs updated: DISASTER-RECOVERY (PITR restore replaced, git RPO,
incident write-up), ARCHITECTURE, LESSONS-LEARNED,
GCP-COST-OPTIMIZATION, TROUBLESHOOTING.

**Kept:** the `audio-separator` GPU service. Its request logs show ~40
external `/separate` calls/month from remote-CLI users. It's already
min-instances 0.

## Already applied to prod (before merge)
- `gcloud run services update karaoke-backend --clear-vpc-connector
--min-instances 1`, same image. Health is OK, and
`/api/health/flacfetch` and `/api/health/encoding-worker` are both OK
without the connector.
- Targeted `pulumi up` on the Firestore DB, both buckets, both jobs and
both 5-min schedulers.
- backup-to-aws redeployed with `deploy.sh`. A manual `?mode=git_repos`
run bundled 13 of 121 repos (106 unchanged, 2 skipped) and uploaded 120
files with 0 errors.
- Deleted `gs://…/backups/flacfetch-torrent-backup-20260122-1931.tar.gz`
(1 GB).

## After merge
Targeted `pulumi up` for `error-monitor-scheduler`, which must wait for
the image with LOOKBACK=60, then delete the connector and drop the
`vpcaccess` API service from state (`disable_on_destroy=False`, so the
API itself stays enabled).

## Tests
`make test` passes locally. The backup_to_aws function tests (43,
including new incremental, list-based-existence and gcs_sync tests) were
run separately because CI doesn't run them.

Related: nomadkaraoke/karaoke-decide#140 (ARCHIVE lifecycle on
`nomadkaraoke-data`, already applied).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Updates**
* Pending-render retries and stuck-download recovery now run every 10
minutes instead of every 5 minutes.
  * Error monitoring now runs hourly and checks a 60-minute window.
* Cloud Run candidate deployments use one minimum instance instead of
two and no longer use a VPC connector.

* **Backups and Recovery**
* Git repository backups now run weekly, with monthly full refreshes and
incremental updates between them.
* Firestore point-in-time recovery is disabled; recovery uses nightly
exports.
  * Storage buckets now use automatic storage-class management.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Opus 5.5 authored and GitHub committed 2026-09-26 22:59:06 -04:00
1 parent 7179e2863f
commit 8cbcc03099
34 files changed
+780 -232

No files matched your search

+7 -6
View File
@@ -2032,9 +2032,11 @@ jobs:
# that but overshot once the loop was made non-blocking: every burst
# forced scale-out, and each overflow request waited out a ~16s app
# cold start (2026-09-22 16-tab test: warm instance served in 1.2s,
# freshly-spawned ones in 16-19s). 32 + --min-instances 2 gives 64
# warm slots so a 16-tab reload is absorbed without cold starts,
# while staying far below the wedge-era 80.
# freshly-spawned ones in 16-19s). 32 + --min-instances 2 gave 64
# warm slots; min-instances was cut to 1 on 2026-09-26 (GCP cost cut,
# accepted perf trade-off: big bursts may hit a ~15s cold start) while
# staying far below the wedge-era 80. --clear-vpc-connector: the
# serverless VPC connector was deleted (flacfetch is public now).
gcloud run deploy karaoke-backend \
--image $IMAGE \
--no-traffic \
@@ -2050,9 +2052,8 @@ jobs:
--timeout 1800 \
--concurrency 32 \
--max-instances 20 \
--min-instances 2 \
--vpc-connector projects/nomadkaraoke/locations/us-central1/connectors/cloud-run-connector \
--vpc-egress private-ranges-only \
--min-instances 1 \
--clear-vpc-connector \
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,DEFAULT_PRIVATE_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-NonPublished,DEFAULT_PRIVATE_BRAND_PREFIX=NOMADNP,USE_GCE_ENCODING=true,USE_GCE_PREVIEW_ENCODING=true,ENABLE_PUSH_NOTIFICATIONS=true,USE_CLOUD_RUN_JOBS_FOR_VIDEO=true,USE_CLOUD_RUN_JOBS_FOR_RENDER=true,EMAIL_PROVIDER=postmark,AUTO_CORRECT_MODEL=claude-opus-5-5,AUTO_CORRECT_COMPARE_MODELS=claude-opus-5-5;gemini-3.8-flash,AUTO_CORRECT_PROACTIVE_ENABLED=true,AUTO_APPROVAL_BACKING_KEEP_ENABLED=true,COMMUNITY_DAILY_PICK_ENABLED=true,EDGE_AUTH_MODE=enforce,COMMIT_SHA=${{ steps.build-meta.outputs.sha }},PR_NUMBER=${{ steps.build-meta.outputs.pr_number }},PR_TITLE=${{ steps.build-meta.outputs.pr_title }}" \
--set-secrets "ANTHROPIC_API_KEY=anthropic-api-key:latest,AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest,POSTMARK_SERVER_TOKEN=postmark-server-token:latest,STRIPE_SECRET_KEY=stripe-secret-key:latest,STRIPE_WEBHOOK_SECRET=stripe-webhook-secret:latest,LANGFUSE_PUBLIC_KEY=langfuse-public-key:latest,LANGFUSE_SECRET_KEY=langfuse-secret-key:latest,LANGFUSE_HOST=langfuse-host:latest,ENCODING_WORKER_URL=encoding-worker-url:latest,ENCODING_WORKER_API_KEY=encoding-worker-api-key:latest,ENCODING_WORKER_FALLBACK_VMS=encoding-worker-fallback-vms:latest,VAPID_PUBLIC_KEY=vapid-public-key:latest,VAPID_PRIVATE_KEY=vapid-private-key:latest,E2E_BYPASS_KEY=e2e-bypass-key:latest,GDRIVE_VALIDATOR_URL=gdrive-validator-url:latest,EDGE_ORIGIN_SECRET=edge-origin-secret:latest,KARAOKEHUNT_FORWARDER_SECRET=karaokehunt-forwarder-secret:latest,KJBOX_PARTNER_SECRET=kjbox-partner-secret:latest"
+21 -4
View File
@@ -55,21 +55,25 @@ jobs:
BUCKET = os.environ["BUCKET"]
# (label, prefix, max_age_hours, require_suffix)
# require_suffix: only objects with this key suffix count toward
# freshness (None = any object). git-repos/ requires ".bundle" so the
# always-written manifest.json can't mark the prefix fresh when every
# bundle failed / nothing was bundled — 0 bundles then reads as stale.
# freshness (None = any object).
# NOTE: Firestore and BigQuery export to GCS staging nightly but are
# only uploaded to S3 *weekly* (Sundays) to cut cross-cloud egress —
# see backup_to_aws/main.py (firestore_to_s3_today). This monitor only
# sees the S3 (off-site) copy, so their limits must allow a full week
# plus slack (192h ≈ 8 days), NOT the 36h nightly cadence. GCS job
# files and secrets DO upload to S3 nightly, so they stay at 36h.
# Git repos are also weekly (Sundays) AND incremental since
# 2026-09-26 — only repos pushed that week get a fresh bundle, so a
# quiet week legitimately leaves every .bundle older than 8 days.
# Freshness is judged on git-repos/manifest.json instead (rewritten
# every weekly run), and the manifest's contents are checked below so
# a run where every repo failed still reads as stale.
TARGETS = [
("Firestore", "firestore/", 192, None),
("GCS job files", "gcs/job-files/", 36, None),
("Encrypted secrets", "secrets/", 36, None),
("BigQuery weekly", "bigquery/daily-refresh/", 192, None),
("Git repos", "git-repos/", 36, ".bundle"),
("Git repos", "git-repos/manifest.json", 192, None),
]
s3 = boto3.client("s3")
@@ -97,6 +101,19 @@ jobs:
line = f"{label}: {age_h}h old (limit {max_hours}h) — last {latest.isoformat()}"
(stale if age > max_hours * 3600 else ok).append(line)
# Git manifest contents: ANY bundle error alerts, even though the
# manifest itself is fresh. The git step is weekly + incremental, so
# a repo that failed keeps an ever-older S3 bundle until the next
# Sunday — silence here would let it drift past the 7-day RPO.
try:
import json as _json
m = _json.loads(s3.get_object(Bucket=BUCKET, Key="git-repos/manifest.json")["Body"].read())
if int(m.get("errors", 0)) > 0:
failed = [r.get("full_name") for r in m.get("repos", []) if r.get("status") == "error"]
stale.append(f"Git repos: last run had {m.get('errors')} bundle errors: {', '.join(failed[:10])}")
except Exception as e: # noqa: BLE001
stale.append(f"Git repos: could not read manifest.json ({e})")
summary = ["## DR backup freshness", ""]
if stale:
summary.append("### Stale")