Files
Andrew Beveridge ☄️andClaude Opus 5.5 8cbcc03099 infra: GCP cost cuts (VPC connector, min-instances, Autoclass, weekly incremental git backup, hourly error monitor, PITR off) (#1063)
## Summary
These are the cost cuts Andrew approved to get the whole GCP project
under $300/mo. Performance compromises were accepted.

- **VPC connector removed.** `cloud-run-connector` only existed to reach
the old flacfetch GCE VM's private IP. flacfetch now runs on Netcup
behind a public HTTPS URL. I checked every URL secret the backend uses
and none are private IPs. The backend is detached with
`--clear-vpc-connector`: just dropping the `--vpc-connector` flag from
`gcloud run deploy` would keep the old setting. `audio-download-job` and
`bulk-search-job` no longer set `vpc_access`. Saves ~$12/mo plus the
Network Intelligence resource-hours.
- **`karaoke-backend` min-instances 2 → 1** (ci.yml).
- **Autoclass** on `karaoke-gen-storage-nomadkaraoke` (terminal
NEARLINE, because customers re-download old jobs) and on
`nomadkaraoke-divebar-files` (terminal ARCHIVE). Autoclass charges no
retrieval or early-deletion fees.
- **backup-to-aws**
- **Git bundles:** now weekly (Sundays) and incremental. A repo is
re-bundled only when its GitHub `pushed_at` is newer than its bundle's
S3 `LastModified` minus a 1h lag, so S3 itself is the state store. Every
repo is re-bundled on the first Sunday of each month.
`?mode=git_repos[&full=1]` runs the git step alone.
- **Bug fix:** since #997 every S3 **overwrite** has failed. The
`backup-writer` IAM user has no `s3:GetObject`, so HEAD on an existing
key returns 403. Git bundles and the manifest were stuck at 09-15, and
the DR freshness monitor has been red every day. The existence check now
uses `ListObjectsV2`.
- **`gcs_sync`:** detects changes by `time_created` instead of
`updated`. Autoclass transitions bump `updated`, which would have made
the sync re-copy every object and pull it back to Standard.
- **DR freshness monitor:** `git-repos/` is now judged on
`manifest.json` (192h) plus a check of the manifest's contents.
- **Error monitor:** runs hourly instead of every 15 min. Lookback is 60
min, equal to the interval, so there are no gaps or double counts.
`MAX_LOG_ENTRIES` goes from 500 to 2000.
- **Firestore PITR disabled.** The nightly export to GCS staging
remains, plus the weekly copy to S3.
- **`recover-stuck-downloads` / `retry-pending-render-jobs`:** every 5 →
10 min. Both are recovery sweeps with ~10-min stall thresholds.
- Docs updated: DISASTER-RECOVERY (PITR restore replaced, git RPO,
incident write-up), ARCHITECTURE, LESSONS-LEARNED,
GCP-COST-OPTIMIZATION, TROUBLESHOOTING.

**Kept:** the `audio-separator` GPU service. Its request logs show ~40
external `/separate` calls/month from remote-CLI users. It's already
min-instances 0.

## Already applied to prod (before merge)
- `gcloud run services update karaoke-backend --clear-vpc-connector
--min-instances 1`, same image. Health is OK, and
`/api/health/flacfetch` and `/api/health/encoding-worker` are both OK
without the connector.
- Targeted `pulumi up` on the Firestore DB, both buckets, both jobs and
both 5-min schedulers.
- backup-to-aws redeployed with `deploy.sh`. A manual `?mode=git_repos`
run bundled 13 of 121 repos (106 unchanged, 2 skipped) and uploaded 120
files with 0 errors.
- Deleted `gs://…/backups/flacfetch-torrent-backup-20260122-1931.tar.gz`
(1 GB).

## After merge
Targeted `pulumi up` for `error-monitor-scheduler`, which must wait for
the image with LOOKBACK=60, then delete the connector and drop the
`vpcaccess` API service from state (`disable_on_destroy=False`, so the
API itself stays enabled).

## Tests
`make test` passes locally. The backup_to_aws function tests (43,
including new incremental, list-based-existence and gcs_sync tests) were
run separately because CI doesn't run them.

Related: nomadkaraoke/karaoke-decide#140 (ARCHIVE lifecycle on
`nomadkaraoke-data`, already applied).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Updates**
* Pending-render retries and stuck-download recovery now run every 10
minutes instead of every 5 minutes.
  * Error monitoring now runs hourly and checks a 60-minute window.
* Cloud Run candidate deployments use one minimum instance instead of
two and no longer use a VPC connector.

* **Backups and Recovery**
* Git repository backups now run weekly, with monthly full refreshes and
incremental updates between them.
* Firestore point-in-time recovery is disabled; recovery uses nightly
exports.
  * Storage buckets now use automatic storage-class management.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-26 22:59:06 -04:00

337 lines
14 KiB
Python

"""
Firestore database resources.
Manages the Firestore database, composite indexes, and TTL configuration.
"""
import pulumi
import pulumi_gcp as gcp
from pulumi_gcp import firestore
from config import PROJECT_ID
def create_database() -> dict:
"""
Create Firestore database and all composite indexes.
Returns:
dict: Dictionary containing all created resources for export.
"""
resources = {}
# ==================== Firestore Database ====================
firestore_db = firestore.Database(
"karaoke-firestore",
project=PROJECT_ID,
name="(default)",
location_id="us-central1",
type="FIRESTORE_NATIVE",
concurrency_mode="PESSIMISTIC",
app_engine_integration_mode="DISABLED",
# PITR disabled 2026-09-26 (cost cut): the backup-to-aws function exports
# Firestore nightly to GCS staging (1-day local RPO) and weekly to S3.
point_in_time_recovery_enablement="POINT_IN_TIME_RECOVERY_DISABLED",
)
resources["firestore_db"] = firestore_db
# ==================== Firestore Composite Indexes ====================
# These are required for queries that filter/sort on multiple fields.
# Jobs: query by user_email, order by created_at (for user job lists)
resources["firestore_index_jobs_user_email"] = firestore.Index(
"firestore-index-jobs-user-email",
project=PROJECT_ID,
database=firestore_db.name,
collection="jobs",
fields=[
firestore.IndexFieldArgs(field_path="user_email", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
],
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# Jobs: query by status, order by created_at (for admin/system job lists)
resources["firestore_index_jobs_status"] = firestore.Index(
"firestore-index-jobs-status",
project=PROJECT_ID,
database=firestore_db.name,
collection="jobs",
fields=[
firestore.IndexFieldArgs(field_path="status", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
],
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# Jobs: query by tenant_id, order by created_at (for tenant portal job lists)
resources["firestore_index_jobs_tenant"] = firestore.Index(
"firestore-index-jobs-tenant",
project=PROJECT_ID,
database=firestore_db.name,
collection="jobs",
fields=[
firestore.IndexFieldArgs(field_path="tenant_id", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
],
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# Jobs: query by user_email + tenant_id, order by created_at (for tenant user job lists)
resources["firestore_index_jobs_user_tenant"] = firestore.Index(
"firestore-index-jobs-user-tenant",
project=PROJECT_ID,
database=firestore_db.name,
collection="jobs",
fields=[
firestore.IndexFieldArgs(field_path="user_email", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="tenant_id", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
],
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# Jobs: query by customer_email + made_for_you, order by created_at
# Used for finding existing made-for-you orders for a customer
resources["firestore_index_jobs_customer_mfy"] = firestore.Index(
"firestore-index-jobs-customer-mfy",
project=PROJECT_ID,
database=firestore_db.name,
collection="jobs",
fields=[
firestore.IndexFieldArgs(field_path="customer_email", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="made_for_you", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
],
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# Jobs: query by request_metadata.client_id + tenant_id, order by created_at.
# Powers GET /api/jobs?client_id= — the idempotency lookup for externally
# created jobs (the Fiverr agent tags jobs client_id="fiverr:{order_id}").
# Without it that query 400s ("The query requires an index").
resources["firestore_index_jobs_client_id"] = firestore.Index(
"firestore-index-jobs-client-id",
project=PROJECT_ID,
database=firestore_db.name,
collection="jobs",
fields=[
firestore.IndexFieldArgs(field_path="request_metadata.client_id", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="tenant_id", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
],
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# Sessions: query by user_email + is_active, order by created_at
resources["firestore_index_sessions_active"] = firestore.Index(
"firestore-index-sessions-active",
project=PROJECT_ID,
database=firestore_db.name,
collection="sessions",
fields=[
firestore.IndexFieldArgs(field_path="is_active", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="user_email", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
],
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# Magic links: query by email, order by created_at
resources["firestore_index_magic_links"] = firestore.Index(
"firestore-index-magic-links",
project=PROJECT_ID,
database=firestore_db.name,
collection="magic_links",
fields=[
firestore.IndexFieldArgs(field_path="email", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
],
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# Gen Users: query by is_active, order by created_at
resources["firestore_index_gen_users_active_created"] = firestore.Index(
"firestore-index-gen-users-active-created",
project=PROJECT_ID,
database=firestore_db.name,
collection="gen_users",
fields=[
firestore.IndexFieldArgs(field_path="is_active", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
],
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# Gen Users: query by is_active, order by last_login_at
resources["firestore_index_gen_users_active_login"] = firestore.Index(
"firestore-index-gen-users-active-login",
project=PROJECT_ID,
database=firestore_db.name,
collection="gen_users",
fields=[
firestore.IndexFieldArgs(field_path="is_active", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="last_login_at", order="DESCENDING"),
],
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# Gen Users: query by is_active, order by credits
resources["firestore_index_gen_users_active_credits"] = firestore.Index(
"firestore-index-gen-users-active-credits",
project=PROJECT_ID,
database=firestore_db.name,
collection="gen_users",
fields=[
firestore.IndexFieldArgs(field_path="is_active", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="credits", order="DESCENDING"),
],
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# Gen Users: query by is_active, order by email
resources["firestore_index_gen_users_active_email"] = firestore.Index(
"firestore-index-gen-users-active-email",
project=PROJECT_ID,
database=firestore_db.name,
collection="gen_users",
fields=[
firestore.IndexFieldArgs(field_path="is_active", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="email", order="ASCENDING"),
],
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# Gen Users: query by signup_ip, order by created_at (anti-abuse investigation)
resources["firestore_index_gen_users_signup_ip"] = firestore.Index(
"firestore-index-gen-users-signup-ip",
project=PROJECT_ID,
database=firestore_db.name,
collection="gen_users",
fields=[
firestore.IndexFieldArgs(field_path="signup_ip", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
],
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# Gen Users: query by device_fingerprint, order by created_at (anti-abuse investigation)
resources["firestore_index_gen_users_device_fingerprint"] = firestore.Index(
"firestore-index-gen-users-device-fingerprint",
project=PROJECT_ID,
database=firestore_db.name,
collection="gen_users",
fields=[
firestore.IndexFieldArgs(field_path="device_fingerprint", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
],
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# Stripe Payments: query by customer_email, order by created_at
# Used for admin user detail page payment history
resources["firestore_index_stripe_payments_email"] = firestore.Index(
"firestore-index-stripe-payments-email",
project=PROJECT_ID,
database=firestore_db.name,
collection="stripe_payments",
fields=[
firestore.IndexFieldArgs(field_path="customer_email", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
],
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# Email log: query by recipient, order by created_at
# Used for the admin user-detail "Emails sent to this user" history view
resources["firestore_index_email_log_recipient"] = firestore.Index(
"firestore-index-email-log-recipient",
project=PROJECT_ID,
database=firestore_db.name,
collection="email_log",
fields=[
firestore.IndexFieldArgs(field_path="recipient", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
],
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# Audio separation jobs: query by status + updated_at (for cleanup of old jobs)
resources["firestore_index_audio_separation_jobs_cleanup"] = firestore.Index(
"firestore-index-audio-separation-jobs-cleanup",
project=PROJECT_ID,
database=firestore_db.name,
collection="audio_separation_jobs",
fields=[
firestore.IndexFieldArgs(field_path="status", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="updated_at", order="ASCENDING"),
],
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# ==================== Firestore TTL Field Configuration ====================
# TTL policies automatically delete documents after expiration.
# Worker logs TTL: logs subcollection documents expire after 30 days
resources["firestore_field_logs_ttl"] = firestore.Field(
"firestore-field-logs-ttl",
project=PROJECT_ID,
database=firestore_db.name,
collection="logs", # Subcollection name (applies to all jobs/{job_id}/logs)
field="ttl_expiry",
ttl_config={}, # Empty block enables TTL based on ttl_expiry field
index_config={}, # Disable indexing on TTL field
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# Search sessions TTL: short-lived sessions from guided job creation flow expire after 30 minutes
# Sessions are created by POST /api/audio-search/search-standalone and consumed by
# POST /api/jobs/create-from-search. The TTL is a safety net for abandoned sessions.
resources["firestore_field_search_sessions_ttl"] = firestore.Field(
"firestore-field-search-sessions-ttl",
project=PROJECT_ID,
database=firestore_db.name,
collection="search_sessions",
field="ttl_expiry",
ttl_config={}, # Empty block enables TTL based on ttl_expiry field
index_config={}, # Disable indexing on TTL field
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
# ==================== Review Sessions Collection Group Index ====================
# The cross-job session search uses a collection group query on
# review_sessions ordered by updated_at DESC. Firestore requires a
# single-field index exemption with COLLECTION_GROUP scope for this.
# Created via REST API (gcloud doesn't support query-scope for field exemptions):
#
# curl -X PATCH \
# "https://firestore.googleapis.com/v1/projects/nomadkaraoke/databases/(default)/collectionGroups/review_sessions/fields/updated_at" \
# -H "Authorization: Bearer $(gcloud auth print-access-token)" \
# -H "Content-Type: application/json" \
# -d '{"indexConfig":{"indexes":[
# {"queryScope":"COLLECTION","fields":[{"fieldPath":"updated_at","order":"ASCENDING"}]},
# {"queryScope":"COLLECTION","fields":[{"fieldPath":"updated_at","order":"DESCENDING"}]},
# {"queryScope":"COLLECTION_GROUP","fields":[{"fieldPath":"updated_at","order":"DESCENDING"}]}
# ]}}'
#
# Pulumi doesn't have a native resource for single-field index
# exemptions — manage via gcloud/REST if it needs recreation.
# ==================== Firestore Indexes for Logs Subcollection ====================
# Logs: query by worker, order by timestamp
resources["firestore_index_logs_worker_timestamp"] = firestore.Index(
"firestore-index-logs-worker-timestamp",
project=PROJECT_ID,
database=firestore_db.name,
collection="logs", # Subcollection name
fields=[
firestore.IndexFieldArgs(field_path="worker", order="ASCENDING"),
firestore.IndexFieldArgs(field_path="timestamp", order="ASCENDING"),
],
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
)
return resources