Files
Andrew Beveridge ☄️andClaude Opus 5.5 8cbcc03099 infra: GCP cost cuts (VPC connector, min-instances, Autoclass, weekly incremental git backup, hourly error monitor, PITR off) (#1063)
## Summary
These are the cost cuts Andrew approved to get the whole GCP project
under $300/mo. Performance compromises were accepted.

- **VPC connector removed.** `cloud-run-connector` only existed to reach
the old flacfetch GCE VM's private IP. flacfetch now runs on Netcup
behind a public HTTPS URL. I checked every URL secret the backend uses
and none are private IPs. The backend is detached with
`--clear-vpc-connector`: just dropping the `--vpc-connector` flag from
`gcloud run deploy` would keep the old setting. `audio-download-job` and
`bulk-search-job` no longer set `vpc_access`. Saves ~$12/mo plus the
Network Intelligence resource-hours.
- **`karaoke-backend` min-instances 2 → 1** (ci.yml).
- **Autoclass** on `karaoke-gen-storage-nomadkaraoke` (terminal
NEARLINE, because customers re-download old jobs) and on
`nomadkaraoke-divebar-files` (terminal ARCHIVE). Autoclass charges no
retrieval or early-deletion fees.
- **backup-to-aws**
- **Git bundles:** now weekly (Sundays) and incremental. A repo is
re-bundled only when its GitHub `pushed_at` is newer than its bundle's
S3 `LastModified` minus a 1h lag, so S3 itself is the state store. Every
repo is re-bundled on the first Sunday of each month.
`?mode=git_repos[&full=1]` runs the git step alone.
- **Bug fix:** since #997 every S3 **overwrite** has failed. The
`backup-writer` IAM user has no `s3:GetObject`, so HEAD on an existing
key returns 403. Git bundles and the manifest were stuck at 09-15, and
the DR freshness monitor has been red every day. The existence check now
uses `ListObjectsV2`.
- **`gcs_sync`:** detects changes by `time_created` instead of
`updated`. Autoclass transitions bump `updated`, which would have made
the sync re-copy every object and pull it back to Standard.
- **DR freshness monitor:** `git-repos/` is now judged on
`manifest.json` (192h) plus a check of the manifest's contents.
- **Error monitor:** runs hourly instead of every 15 min. Lookback is 60
min, equal to the interval, so there are no gaps or double counts.
`MAX_LOG_ENTRIES` goes from 500 to 2000.
- **Firestore PITR disabled.** The nightly export to GCS staging
remains, plus the weekly copy to S3.
- **`recover-stuck-downloads` / `retry-pending-render-jobs`:** every 5 →
10 min. Both are recovery sweeps with ~10-min stall thresholds.
- Docs updated: DISASTER-RECOVERY (PITR restore replaced, git RPO,
incident write-up), ARCHITECTURE, LESSONS-LEARNED,
GCP-COST-OPTIMIZATION, TROUBLESHOOTING.

**Kept:** the `audio-separator` GPU service. Its request logs show ~40
external `/separate` calls/month from remote-CLI users. It's already
min-instances 0.

## Already applied to prod (before merge)
- `gcloud run services update karaoke-backend --clear-vpc-connector
--min-instances 1`, same image. Health is OK, and
`/api/health/flacfetch` and `/api/health/encoding-worker` are both OK
without the connector.
- Targeted `pulumi up` on the Firestore DB, both buckets, both jobs and
both 5-min schedulers.
- backup-to-aws redeployed with `deploy.sh`. A manual `?mode=git_repos`
run bundled 13 of 121 repos (106 unchanged, 2 skipped) and uploaded 120
files with 0 errors.
- Deleted `gs://…/backups/flacfetch-torrent-backup-20260122-1931.tar.gz`
(1 GB).

## After merge
Targeted `pulumi up` for `error-monitor-scheduler`, which must wait for
the image with LOOKBACK=60, then delete the connector and drop the
`vpcaccess` API service from state (`disable_on_destroy=False`, so the
API itself stays enabled).

## Tests
`make test` passes locally. The backup_to_aws function tests (43,
including new incremental, list-based-existence and gcs_sync tests) were
run separately because CI doesn't run them.

Related: nomadkaraoke/karaoke-decide#140 (ARCHIVE lifecycle on
`nomadkaraoke-data`, already applied).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Updates**
* Pending-render retries and stuck-download recovery now run every 10
minutes instead of every 5 minutes.
  * Error monitoring now runs hourly and checks a 60-minute window.
* Cloud Run candidate deployments use one minimum instance instead of
two and no longer use a VPC connector.

* **Backups and Recovery**
* Git repository backups now run weekly, with monthly full refreshes and
incremental updates between them.
* Firestore point-in-time recovery is disabled; recovery uses nightly
exports.
  * Storage buckets now use automatic storage-class management.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-26 22:59:06 -04:00

28 lines
520 B
Python

"""
Infrastructure modules for Pulumi resources.
Each module is responsible for a specific category of cloud resources.
"""
from . import database
from . import storage
from . import artifact_registry
from . import secrets
from . import cloud_tasks
from . import cloud_run
from . import monitoring
from . import iam
from . import runner_manager
__all__ = [
"database",
"storage",
"artifact_registry",
"secrets",
"cloud_tasks",
"cloud_run",
"monitoring",
"iam",
"runner_manager",
]