mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary These are the cost cuts Andrew approved to get the whole GCP project under $300/mo. Performance compromises were accepted. - **VPC connector removed.** `cloud-run-connector` only existed to reach the old flacfetch GCE VM's private IP. flacfetch now runs on Netcup behind a public HTTPS URL. I checked every URL secret the backend uses and none are private IPs. The backend is detached with `--clear-vpc-connector`: just dropping the `--vpc-connector` flag from `gcloud run deploy` would keep the old setting. `audio-download-job` and `bulk-search-job` no longer set `vpc_access`. Saves ~$12/mo plus the Network Intelligence resource-hours. - **`karaoke-backend` min-instances 2 → 1** (ci.yml). - **Autoclass** on `karaoke-gen-storage-nomadkaraoke` (terminal NEARLINE, because customers re-download old jobs) and on `nomadkaraoke-divebar-files` (terminal ARCHIVE). Autoclass charges no retrieval or early-deletion fees. - **backup-to-aws** - **Git bundles:** now weekly (Sundays) and incremental. A repo is re-bundled only when its GitHub `pushed_at` is newer than its bundle's S3 `LastModified` minus a 1h lag, so S3 itself is the state store. Every repo is re-bundled on the first Sunday of each month. `?mode=git_repos[&full=1]` runs the git step alone. - **Bug fix:** since #997 every S3 **overwrite** has failed. The `backup-writer` IAM user has no `s3:GetObject`, so HEAD on an existing key returns 403. Git bundles and the manifest were stuck at 09-15, and the DR freshness monitor has been red every day. The existence check now uses `ListObjectsV2`. - **`gcs_sync`:** detects changes by `time_created` instead of `updated`. Autoclass transitions bump `updated`, which would have made the sync re-copy every object and pull it back to Standard. - **DR freshness monitor:** `git-repos/` is now judged on `manifest.json` (192h) plus a check of the manifest's contents. - **Error monitor:** runs hourly instead of every 15 min. Lookback is 60 min, equal to the interval, so there are no gaps or double counts. `MAX_LOG_ENTRIES` goes from 500 to 2000. - **Firestore PITR disabled.** The nightly export to GCS staging remains, plus the weekly copy to S3. - **`recover-stuck-downloads` / `retry-pending-render-jobs`:** every 5 → 10 min. Both are recovery sweeps with ~10-min stall thresholds. - Docs updated: DISASTER-RECOVERY (PITR restore replaced, git RPO, incident write-up), ARCHITECTURE, LESSONS-LEARNED, GCP-COST-OPTIMIZATION, TROUBLESHOOTING. **Kept:** the `audio-separator` GPU service. Its request logs show ~40 external `/separate` calls/month from remote-CLI users. It's already min-instances 0. ## Already applied to prod (before merge) - `gcloud run services update karaoke-backend --clear-vpc-connector --min-instances 1`, same image. Health is OK, and `/api/health/flacfetch` and `/api/health/encoding-worker` are both OK without the connector. - Targeted `pulumi up` on the Firestore DB, both buckets, both jobs and both 5-min schedulers. - backup-to-aws redeployed with `deploy.sh`. A manual `?mode=git_repos` run bundled 13 of 121 repos (106 unchanged, 2 skipped) and uploaded 120 files with 0 errors. - Deleted `gs://…/backups/flacfetch-torrent-backup-20260122-1931.tar.gz` (1 GB). ## After merge Targeted `pulumi up` for `error-monitor-scheduler`, which must wait for the image with LOOKBACK=60, then delete the connector and drop the `vpcaccess` API service from state (`disable_on_destroy=False`, so the API itself stays enabled). ## Tests `make test` passes locally. The backup_to_aws function tests (43, including new incremental, list-based-existence and gcs_sync tests) were run separately because CI doesn't run them. Related: nomadkaraoke/karaoke-decide#140 (ARCHIVE lifecycle on `nomadkaraoke-data`, already applied). 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Updates** * Pending-render retries and stuck-download recovery now run every 10 minutes instead of every 5 minutes. * Error monitoring now runs hourly and checks a 60-minute window. * Cloud Run candidate deployments use one minimum instance instead of two and no longer use a VPC connector. * **Backups and Recovery** * Git repository backups now run weekly, with monthly full refreshes and incremental updates between them. * Firestore point-in-time recovery is disabled; recovery uses nightly exports. * Storage buckets now use automatic storage-class management. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
337 lines
14 KiB
Python
337 lines
14 KiB
Python
"""
|
|
Firestore database resources.
|
|
|
|
Manages the Firestore database, composite indexes, and TTL configuration.
|
|
"""
|
|
|
|
import pulumi
|
|
import pulumi_gcp as gcp
|
|
from pulumi_gcp import firestore
|
|
|
|
from config import PROJECT_ID
|
|
|
|
|
|
def create_database() -> dict:
|
|
"""
|
|
Create Firestore database and all composite indexes.
|
|
|
|
Returns:
|
|
dict: Dictionary containing all created resources for export.
|
|
"""
|
|
resources = {}
|
|
|
|
# ==================== Firestore Database ====================
|
|
firestore_db = firestore.Database(
|
|
"karaoke-firestore",
|
|
project=PROJECT_ID,
|
|
name="(default)",
|
|
location_id="us-central1",
|
|
type="FIRESTORE_NATIVE",
|
|
concurrency_mode="PESSIMISTIC",
|
|
app_engine_integration_mode="DISABLED",
|
|
# PITR disabled 2026-09-26 (cost cut): the backup-to-aws function exports
|
|
# Firestore nightly to GCS staging (1-day local RPO) and weekly to S3.
|
|
point_in_time_recovery_enablement="POINT_IN_TIME_RECOVERY_DISABLED",
|
|
)
|
|
resources["firestore_db"] = firestore_db
|
|
|
|
# ==================== Firestore Composite Indexes ====================
|
|
# These are required for queries that filter/sort on multiple fields.
|
|
|
|
# Jobs: query by user_email, order by created_at (for user job lists)
|
|
resources["firestore_index_jobs_user_email"] = firestore.Index(
|
|
"firestore-index-jobs-user-email",
|
|
project=PROJECT_ID,
|
|
database=firestore_db.name,
|
|
collection="jobs",
|
|
fields=[
|
|
firestore.IndexFieldArgs(field_path="user_email", order="ASCENDING"),
|
|
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
|
|
],
|
|
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
|
)
|
|
|
|
# Jobs: query by status, order by created_at (for admin/system job lists)
|
|
resources["firestore_index_jobs_status"] = firestore.Index(
|
|
"firestore-index-jobs-status",
|
|
project=PROJECT_ID,
|
|
database=firestore_db.name,
|
|
collection="jobs",
|
|
fields=[
|
|
firestore.IndexFieldArgs(field_path="status", order="ASCENDING"),
|
|
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
|
|
],
|
|
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
|
)
|
|
|
|
# Jobs: query by tenant_id, order by created_at (for tenant portal job lists)
|
|
resources["firestore_index_jobs_tenant"] = firestore.Index(
|
|
"firestore-index-jobs-tenant",
|
|
project=PROJECT_ID,
|
|
database=firestore_db.name,
|
|
collection="jobs",
|
|
fields=[
|
|
firestore.IndexFieldArgs(field_path="tenant_id", order="ASCENDING"),
|
|
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
|
|
],
|
|
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
|
)
|
|
|
|
# Jobs: query by user_email + tenant_id, order by created_at (for tenant user job lists)
|
|
resources["firestore_index_jobs_user_tenant"] = firestore.Index(
|
|
"firestore-index-jobs-user-tenant",
|
|
project=PROJECT_ID,
|
|
database=firestore_db.name,
|
|
collection="jobs",
|
|
fields=[
|
|
firestore.IndexFieldArgs(field_path="user_email", order="ASCENDING"),
|
|
firestore.IndexFieldArgs(field_path="tenant_id", order="ASCENDING"),
|
|
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
|
|
],
|
|
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
|
)
|
|
|
|
# Jobs: query by customer_email + made_for_you, order by created_at
|
|
# Used for finding existing made-for-you orders for a customer
|
|
resources["firestore_index_jobs_customer_mfy"] = firestore.Index(
|
|
"firestore-index-jobs-customer-mfy",
|
|
project=PROJECT_ID,
|
|
database=firestore_db.name,
|
|
collection="jobs",
|
|
fields=[
|
|
firestore.IndexFieldArgs(field_path="customer_email", order="ASCENDING"),
|
|
firestore.IndexFieldArgs(field_path="made_for_you", order="ASCENDING"),
|
|
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
|
|
],
|
|
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
|
)
|
|
|
|
# Jobs: query by request_metadata.client_id + tenant_id, order by created_at.
|
|
# Powers GET /api/jobs?client_id= — the idempotency lookup for externally
|
|
# created jobs (the Fiverr agent tags jobs client_id="fiverr:{order_id}").
|
|
# Without it that query 400s ("The query requires an index").
|
|
resources["firestore_index_jobs_client_id"] = firestore.Index(
|
|
"firestore-index-jobs-client-id",
|
|
project=PROJECT_ID,
|
|
database=firestore_db.name,
|
|
collection="jobs",
|
|
fields=[
|
|
firestore.IndexFieldArgs(field_path="request_metadata.client_id", order="ASCENDING"),
|
|
firestore.IndexFieldArgs(field_path="tenant_id", order="ASCENDING"),
|
|
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
|
|
],
|
|
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
|
)
|
|
|
|
# Sessions: query by user_email + is_active, order by created_at
|
|
resources["firestore_index_sessions_active"] = firestore.Index(
|
|
"firestore-index-sessions-active",
|
|
project=PROJECT_ID,
|
|
database=firestore_db.name,
|
|
collection="sessions",
|
|
fields=[
|
|
firestore.IndexFieldArgs(field_path="is_active", order="ASCENDING"),
|
|
firestore.IndexFieldArgs(field_path="user_email", order="ASCENDING"),
|
|
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
|
|
],
|
|
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
|
)
|
|
|
|
# Magic links: query by email, order by created_at
|
|
resources["firestore_index_magic_links"] = firestore.Index(
|
|
"firestore-index-magic-links",
|
|
project=PROJECT_ID,
|
|
database=firestore_db.name,
|
|
collection="magic_links",
|
|
fields=[
|
|
firestore.IndexFieldArgs(field_path="email", order="ASCENDING"),
|
|
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
|
|
],
|
|
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
|
)
|
|
|
|
# Gen Users: query by is_active, order by created_at
|
|
resources["firestore_index_gen_users_active_created"] = firestore.Index(
|
|
"firestore-index-gen-users-active-created",
|
|
project=PROJECT_ID,
|
|
database=firestore_db.name,
|
|
collection="gen_users",
|
|
fields=[
|
|
firestore.IndexFieldArgs(field_path="is_active", order="ASCENDING"),
|
|
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
|
|
],
|
|
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
|
)
|
|
|
|
# Gen Users: query by is_active, order by last_login_at
|
|
resources["firestore_index_gen_users_active_login"] = firestore.Index(
|
|
"firestore-index-gen-users-active-login",
|
|
project=PROJECT_ID,
|
|
database=firestore_db.name,
|
|
collection="gen_users",
|
|
fields=[
|
|
firestore.IndexFieldArgs(field_path="is_active", order="ASCENDING"),
|
|
firestore.IndexFieldArgs(field_path="last_login_at", order="DESCENDING"),
|
|
],
|
|
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
|
)
|
|
|
|
# Gen Users: query by is_active, order by credits
|
|
resources["firestore_index_gen_users_active_credits"] = firestore.Index(
|
|
"firestore-index-gen-users-active-credits",
|
|
project=PROJECT_ID,
|
|
database=firestore_db.name,
|
|
collection="gen_users",
|
|
fields=[
|
|
firestore.IndexFieldArgs(field_path="is_active", order="ASCENDING"),
|
|
firestore.IndexFieldArgs(field_path="credits", order="DESCENDING"),
|
|
],
|
|
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
|
)
|
|
|
|
# Gen Users: query by is_active, order by email
|
|
resources["firestore_index_gen_users_active_email"] = firestore.Index(
|
|
"firestore-index-gen-users-active-email",
|
|
project=PROJECT_ID,
|
|
database=firestore_db.name,
|
|
collection="gen_users",
|
|
fields=[
|
|
firestore.IndexFieldArgs(field_path="is_active", order="ASCENDING"),
|
|
firestore.IndexFieldArgs(field_path="email", order="ASCENDING"),
|
|
],
|
|
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
|
)
|
|
|
|
# Gen Users: query by signup_ip, order by created_at (anti-abuse investigation)
|
|
resources["firestore_index_gen_users_signup_ip"] = firestore.Index(
|
|
"firestore-index-gen-users-signup-ip",
|
|
project=PROJECT_ID,
|
|
database=firestore_db.name,
|
|
collection="gen_users",
|
|
fields=[
|
|
firestore.IndexFieldArgs(field_path="signup_ip", order="ASCENDING"),
|
|
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
|
|
],
|
|
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
|
)
|
|
|
|
# Gen Users: query by device_fingerprint, order by created_at (anti-abuse investigation)
|
|
resources["firestore_index_gen_users_device_fingerprint"] = firestore.Index(
|
|
"firestore-index-gen-users-device-fingerprint",
|
|
project=PROJECT_ID,
|
|
database=firestore_db.name,
|
|
collection="gen_users",
|
|
fields=[
|
|
firestore.IndexFieldArgs(field_path="device_fingerprint", order="ASCENDING"),
|
|
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
|
|
],
|
|
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
|
)
|
|
|
|
# Stripe Payments: query by customer_email, order by created_at
|
|
# Used for admin user detail page payment history
|
|
resources["firestore_index_stripe_payments_email"] = firestore.Index(
|
|
"firestore-index-stripe-payments-email",
|
|
project=PROJECT_ID,
|
|
database=firestore_db.name,
|
|
collection="stripe_payments",
|
|
fields=[
|
|
firestore.IndexFieldArgs(field_path="customer_email", order="ASCENDING"),
|
|
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
|
|
],
|
|
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
|
)
|
|
|
|
# Email log: query by recipient, order by created_at
|
|
# Used for the admin user-detail "Emails sent to this user" history view
|
|
resources["firestore_index_email_log_recipient"] = firestore.Index(
|
|
"firestore-index-email-log-recipient",
|
|
project=PROJECT_ID,
|
|
database=firestore_db.name,
|
|
collection="email_log",
|
|
fields=[
|
|
firestore.IndexFieldArgs(field_path="recipient", order="ASCENDING"),
|
|
firestore.IndexFieldArgs(field_path="created_at", order="DESCENDING"),
|
|
],
|
|
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
|
)
|
|
|
|
# Audio separation jobs: query by status + updated_at (for cleanup of old jobs)
|
|
resources["firestore_index_audio_separation_jobs_cleanup"] = firestore.Index(
|
|
"firestore-index-audio-separation-jobs-cleanup",
|
|
project=PROJECT_ID,
|
|
database=firestore_db.name,
|
|
collection="audio_separation_jobs",
|
|
fields=[
|
|
firestore.IndexFieldArgs(field_path="status", order="ASCENDING"),
|
|
firestore.IndexFieldArgs(field_path="updated_at", order="ASCENDING"),
|
|
],
|
|
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
|
)
|
|
|
|
# ==================== Firestore TTL Field Configuration ====================
|
|
# TTL policies automatically delete documents after expiration.
|
|
|
|
# Worker logs TTL: logs subcollection documents expire after 30 days
|
|
resources["firestore_field_logs_ttl"] = firestore.Field(
|
|
"firestore-field-logs-ttl",
|
|
project=PROJECT_ID,
|
|
database=firestore_db.name,
|
|
collection="logs", # Subcollection name (applies to all jobs/{job_id}/logs)
|
|
field="ttl_expiry",
|
|
ttl_config={}, # Empty block enables TTL based on ttl_expiry field
|
|
index_config={}, # Disable indexing on TTL field
|
|
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
|
)
|
|
|
|
# Search sessions TTL: short-lived sessions from guided job creation flow expire after 30 minutes
|
|
# Sessions are created by POST /api/audio-search/search-standalone and consumed by
|
|
# POST /api/jobs/create-from-search. The TTL is a safety net for abandoned sessions.
|
|
resources["firestore_field_search_sessions_ttl"] = firestore.Field(
|
|
"firestore-field-search-sessions-ttl",
|
|
project=PROJECT_ID,
|
|
database=firestore_db.name,
|
|
collection="search_sessions",
|
|
field="ttl_expiry",
|
|
ttl_config={}, # Empty block enables TTL based on ttl_expiry field
|
|
index_config={}, # Disable indexing on TTL field
|
|
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
|
)
|
|
|
|
# ==================== Review Sessions Collection Group Index ====================
|
|
|
|
# The cross-job session search uses a collection group query on
|
|
# review_sessions ordered by updated_at DESC. Firestore requires a
|
|
# single-field index exemption with COLLECTION_GROUP scope for this.
|
|
# Created via REST API (gcloud doesn't support query-scope for field exemptions):
|
|
#
|
|
# curl -X PATCH \
|
|
# "https://firestore.googleapis.com/v1/projects/nomadkaraoke/databases/(default)/collectionGroups/review_sessions/fields/updated_at" \
|
|
# -H "Authorization: Bearer $(gcloud auth print-access-token)" \
|
|
# -H "Content-Type: application/json" \
|
|
# -d '{"indexConfig":{"indexes":[
|
|
# {"queryScope":"COLLECTION","fields":[{"fieldPath":"updated_at","order":"ASCENDING"}]},
|
|
# {"queryScope":"COLLECTION","fields":[{"fieldPath":"updated_at","order":"DESCENDING"}]},
|
|
# {"queryScope":"COLLECTION_GROUP","fields":[{"fieldPath":"updated_at","order":"DESCENDING"}]}
|
|
# ]}}'
|
|
#
|
|
# Pulumi doesn't have a native resource for single-field index
|
|
# exemptions — manage via gcloud/REST if it needs recreation.
|
|
|
|
# ==================== Firestore Indexes for Logs Subcollection ====================
|
|
|
|
# Logs: query by worker, order by timestamp
|
|
resources["firestore_index_logs_worker_timestamp"] = firestore.Index(
|
|
"firestore-index-logs-worker-timestamp",
|
|
project=PROJECT_ID,
|
|
database=firestore_db.name,
|
|
collection="logs", # Subcollection name
|
|
fields=[
|
|
firestore.IndexFieldArgs(field_path="worker", order="ASCENDING"),
|
|
firestore.IndexFieldArgs(field_path="timestamp", order="ASCENDING"),
|
|
],
|
|
opts=pulumi.ResourceOptions(depends_on=[firestore_db]),
|
|
)
|
|
|
|
return resources
|