mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary
Extends the nightly DR backup Cloud Function (`backup-to-aws`) to bundle
**every git repo under `github.com/nomadkaraoke` and
`github.com/beveradb`** into a per-repo `git bundle` on S3, so code +
full history survive a complete loss of GitHub access (e.g. an automated
copyright-strike account ban). Each bundle is directly cloneable —
restore is `git clone --mirror <bundle>` → `git push --mirror
<new-remote>`.
## Changes
- **`git_repos_export.py`** (new): enumerate repos via GitHub
`/user/repos` (includes private), mirror-clone + `git bundle create
--all`, upload to `git-repos/{owner}/{repo}.bundle` in staging, plus a
`manifest.json` inventory. Forks excluded, oversized/empty repos
skipped, per-repo errors non-fatal, org backed up **before** the
personal account for deadline resilience. PAT is read from Secret
Manager at **runtime** (not a deploy-time secret env var) so the
function deploys before the secret has a value and skips gracefully if
absent.
- **`main.py`**: new nightly Step 4b (between secrets export and S3
upload).
- **`infrastructure/modules/backup.py`**: bump function to 4Gi/2CPU
(mirror-clone uses in-memory `/tmp`); add `GIT_BACKUP_OWNERS` env.
- **`infrastructure/modules/secrets.py`**: new `github-backup-token`
secret (value added manually).
- **`.github/workflows/dr-backup-freshness.yml`**: monitor `git-repos/`
at 36h, counting only `.bundle` objects.
- **`docs/DISASTER-RECOVERY.md`**: new "Scenario 3: Loss of GitHub
access" + restore steps + token setup.
## Security
- The GitHub PAT is injected via a `GIT_CONFIG_*` `Authorization` header
(like `actions/checkout`), never in the clone URL/argv — so it can't
leak via `ps`, `TimeoutExpired`, or captured stderr. Token is also
redacted from any error text persisted into `manifest.json`.
## Testing
- 14 new unit tests for the module
(enumerate/filter/bundle/skip/error/security paths); full
`backup_to_aws` suite: **27 passing**.
- CodeRabbit local review (cycle 1) completed; all 4 findings (1
critical PAT leak, 2 major, 1 minor) addressed. Cycle-2 verification was
rate-limited (free-tier), so **left for the GitHub bot to review the
final diff**.
## Deploy notes
- **Pulumi already applied locally** (`github-backup-token` secret
created + function on 4Gi/2CPU) — CI re-run is a no-op.
- After merge, run `infrastructure/functions/backup_to_aws/deploy.sh` to
ship the new function source, then set the PAT:
`printf %s '<PAT>' | gcloud secrets versions add github-backup-token
--data-file=- --project=nomadkaraoke`
- Requires a classic GitHub PAT (owner: beveradb, member of nomadkaraoke
org) with `repo` + `read:org`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---
@coderabbitai ignore
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
289 lines
12 KiB
Python
289 lines
12 KiB
Python
"""
|
|
Backup infrastructure module.
|
|
|
|
Creates:
|
|
- GCS staging bucket for backup exports
|
|
- Service account for backup Cloud Function
|
|
- Cloud Function v2 (Gen2) for nightly backups
|
|
- Cloud Scheduler job for daily trigger
|
|
"""
|
|
|
|
import pulumi
|
|
import pulumi_gcp as gcp
|
|
from pulumi_gcp import (
|
|
cloudfunctionsv2,
|
|
cloudscheduler,
|
|
serviceaccount,
|
|
storage,
|
|
)
|
|
|
|
from config import PROJECT_ID, REGION, get_project_number
|
|
|
|
|
|
def create_backup_resources(all_secrets: dict) -> dict:
|
|
"""Create all backup infrastructure resources."""
|
|
resources = {}
|
|
|
|
# ==================== Service Account ====================
|
|
|
|
sa = serviceaccount.Account(
|
|
"backup-to-aws-sa",
|
|
account_id="backup-to-aws",
|
|
display_name="Backup to AWS Function",
|
|
description="Service account for nightly backup Cloud Function",
|
|
)
|
|
resources["service_account"] = sa
|
|
|
|
# Firestore export permission
|
|
resources["firestore_export"] = gcp.projects.IAMMember(
|
|
"backup-firestore-export",
|
|
project=PROJECT_ID,
|
|
role="roles/datastore.importExportAdmin",
|
|
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
|
|
)
|
|
|
|
# BigQuery read access
|
|
resources["bq_viewer"] = gcp.projects.IAMMember(
|
|
"backup-bigquery-viewer",
|
|
project=PROJECT_ID,
|
|
role="roles/bigquery.dataViewer",
|
|
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
|
|
)
|
|
|
|
resources["bq_job_user"] = gcp.projects.IAMMember(
|
|
"backup-bigquery-job-user",
|
|
project=PROJECT_ID,
|
|
role="roles/bigquery.jobUser",
|
|
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
|
|
)
|
|
|
|
# Secret Manager access (for AWS credentials + nightly secrets backup)
|
|
resources["secret_accessor"] = gcp.projects.IAMMember(
|
|
"backup-secret-accessor",
|
|
project=PROJECT_ID,
|
|
role="roles/secretmanager.secretAccessor",
|
|
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
|
|
)
|
|
|
|
# secretmanager.viewer is required to LIST secrets — accessor only allows reading
|
|
# a known name. Backup needs to enumerate every secret in the project.
|
|
resources["secret_viewer"] = gcp.projects.IAMMember(
|
|
"backup-secret-viewer",
|
|
project=PROJECT_ID,
|
|
role="roles/secretmanager.viewer",
|
|
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
|
|
)
|
|
|
|
# Logging
|
|
resources["logging"] = gcp.projects.IAMMember(
|
|
"backup-logging",
|
|
project=PROJECT_ID,
|
|
role="roles/logging.logWriter",
|
|
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
|
|
)
|
|
|
|
# ==================== Staging Bucket ====================
|
|
|
|
staging_bucket = storage.Bucket(
|
|
"backup-staging-bucket",
|
|
name="nomadkaraoke-backup-staging",
|
|
location="US-CENTRAL1",
|
|
force_destroy=True, # Staging only — OK to destroy
|
|
uniform_bucket_level_access=True,
|
|
lifecycle_rules=[
|
|
storage.BucketLifecycleRuleArgs(
|
|
action=storage.BucketLifecycleRuleActionArgs(type="Delete"),
|
|
condition=storage.BucketLifecycleRuleConditionArgs(age=7),
|
|
),
|
|
],
|
|
)
|
|
resources["staging_bucket"] = staging_bucket
|
|
|
|
# SA needs objectAdmin on staging bucket (read + write + delete)
|
|
resources["staging_bucket_access"] = storage.BucketIAMMember(
|
|
"backup-staging-bucket-access",
|
|
bucket=staging_bucket.name,
|
|
role="roles/storage.objectAdmin",
|
|
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
|
|
)
|
|
|
|
# Firestore service agent needs write access to staging bucket for exports
|
|
resources["firestore_agent_staging_access"] = storage.BucketIAMMember(
|
|
"backup-firestore-agent-staging-access",
|
|
bucket=staging_bucket.name,
|
|
role="roles/storage.objectAdmin",
|
|
member=f"serviceAccount:service-{get_project_number()}@gcp-sa-firestore.iam.gserviceaccount.com",
|
|
)
|
|
|
|
# SA needs objectViewer on source buckets
|
|
source_buckets = [
|
|
"karaoke-gen-storage-nomadkaraoke",
|
|
"nomadkaraoke-kn-data",
|
|
]
|
|
for i, bucket_name in enumerate(source_buckets):
|
|
resources[f"source_bucket_access_{i}"] = storage.BucketIAMMember(
|
|
f"backup-source-bucket-access-{i}",
|
|
bucket=bucket_name,
|
|
role="roles/storage.objectViewer",
|
|
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
|
|
)
|
|
|
|
# ==================== Function Source Bucket ====================
|
|
|
|
function_source_bucket = storage.Bucket(
|
|
"backup-function-source",
|
|
name=f"backup-to-aws-source-{PROJECT_ID}",
|
|
location="US-CENTRAL1",
|
|
force_destroy=True,
|
|
uniform_bucket_level_access=True,
|
|
)
|
|
resources["function_source_bucket"] = function_source_bucket
|
|
|
|
# ==================== Cloud Function ====================
|
|
|
|
function = cloudfunctionsv2.Function(
|
|
"backup-to-aws-function",
|
|
name="backup-to-aws",
|
|
location=REGION,
|
|
description="Nightly backup of Firestore, BigQuery, and GCS to AWS S3",
|
|
build_config=cloudfunctionsv2.FunctionBuildConfigArgs(
|
|
runtime="python312",
|
|
entry_point="backup_to_aws",
|
|
source=cloudfunctionsv2.FunctionBuildConfigSourceArgs(
|
|
storage_source=cloudfunctionsv2.FunctionBuildConfigSourceStorageSourceArgs(
|
|
bucket=function_source_bucket.name,
|
|
object="backup-to-aws-source.zip",
|
|
),
|
|
),
|
|
),
|
|
service_config=cloudfunctionsv2.FunctionServiceConfigArgs(
|
|
# 4Gi/2CPU (up from 2Gi/1CPU): the git-repo backup mirror-clones each
|
|
# repo into the in-memory /tmp (Cloud Run tmpfs counts against
|
|
# memory) before bundling. Repos are processed serially and cleaned
|
|
# up between each, so peak is one repo's working set, but the bump
|
|
# gives headroom for the largest repo + faster clone/pack on 2 CPU.
|
|
available_memory="4Gi",
|
|
available_cpu="2",
|
|
# Kept in lockstep with the scheduler's attempt_deadline (1800s, the
|
|
# Cloud Scheduler max) below. If the function could run longer than
|
|
# the scheduler waits, runs in that gap would re-trigger the very
|
|
# timeout→retry→"Path already exists" collision this aligns to avoid.
|
|
timeout_seconds=1800,
|
|
min_instance_count=0,
|
|
max_instance_count=1,
|
|
service_account_email=sa.email,
|
|
environment_variables={
|
|
"GCP_PROJECT": PROJECT_ID,
|
|
"STAGING_BUCKET": "nomadkaraoke-backup-staging",
|
|
"S3_BUCKET": "nomadkaraoke-backup",
|
|
# Curve25519 public key (hex) for sealed-box encryption of secrets backup.
|
|
# The matching private key lives only in 1Password — function cannot decrypt.
|
|
# Set via: pulumi config set backupEncryptionPubkey <64-char-hex>
|
|
"BACKUP_ENCRYPTION_PUBKEY": pulumi.Config().require("backupEncryptionPubkey"),
|
|
# GitHub owners whose repos get bundled to S3 nightly (comma-sep).
|
|
# Backs up code + full history so it survives a GitHub account ban.
|
|
"GIT_BACKUP_OWNERS": "nomadkaraoke,beveradb",
|
|
},
|
|
secret_environment_variables=[
|
|
cloudfunctionsv2.FunctionServiceConfigSecretEnvironmentVariableArgs(
|
|
key="DISCORD_WEBHOOK_URL",
|
|
project_id=PROJECT_ID,
|
|
secret=all_secrets["discord-alert-webhook"].secret_id,
|
|
version="latest",
|
|
),
|
|
# NOTE: the GitHub backup PAT (secret "github-backup-token") is
|
|
# deliberately NOT wired as a secret env var. A version=latest
|
|
# env var fails the function deploy until the secret has a value,
|
|
# but the secret is created empty (value added manually). Instead
|
|
# the function reads it from Secret Manager at *runtime*
|
|
# (git_repos_export.get_github_token) and skips the git-repo step
|
|
# if it's absent — so deploy never blocks on it. The SA already
|
|
# holds project-wide secretmanager.secretAccessor.
|
|
],
|
|
),
|
|
)
|
|
resources["function"] = function
|
|
|
|
# Allow Cloud Scheduler to invoke the function
|
|
resources["scheduler_invoker"] = cloudfunctionsv2.FunctionIamMember(
|
|
"backup-scheduler-invoker",
|
|
project=PROJECT_ID,
|
|
location=REGION,
|
|
cloud_function=function.name,
|
|
role="roles/cloudfunctions.invoker",
|
|
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
|
|
)
|
|
|
|
# Gen2 functions need Cloud Run invoker too
|
|
resources["run_invoker"] = gcp.cloudrunv2.ServiceIamMember(
|
|
"backup-sa-run-invoker",
|
|
project=PROJECT_ID,
|
|
location=REGION,
|
|
name=function.name,
|
|
role="roles/run.invoker",
|
|
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
|
|
)
|
|
|
|
# ==================== Cloud Scheduler ====================
|
|
|
|
scheduler = cloudscheduler.Job(
|
|
"backup-scheduler",
|
|
name="backup-to-aws-nightly",
|
|
description="Trigger nightly backup to AWS S3",
|
|
region=REGION,
|
|
schedule="0 1 * * *",
|
|
time_zone="America/New_York",
|
|
# The backup pipeline (esp. the Firestore export) runs for several
|
|
# minutes. Without an explicit deadline, Cloud Scheduler defaults to
|
|
# 180s, times out the HTTP call, and retries — while the original
|
|
# invocation keeps running to success. The retry then collides on the
|
|
# already-created firestore/<date>/ export path ("400 Path already
|
|
# exists"), producing a spurious FAILED backup report every night.
|
|
# 1800s (30m, the Scheduler max) lets it wait for the real result.
|
|
attempt_deadline="1800s",
|
|
http_target=cloudscheduler.JobHttpTargetArgs(
|
|
uri=function.url,
|
|
http_method="POST",
|
|
oidc_token=cloudscheduler.JobHttpTargetOidcTokenArgs(
|
|
service_account_email=sa.email,
|
|
),
|
|
),
|
|
retry_config=cloudscheduler.JobRetryConfigArgs(
|
|
retry_count=2,
|
|
min_backoff_duration="300s",
|
|
max_backoff_duration="1800s",
|
|
),
|
|
)
|
|
resources["scheduler"] = scheduler
|
|
|
|
# Quarterly DR restore drill reminder — posts a Discord nudge so the human
|
|
# remembers to run the drill in docs/DISASTER-RECOVERY.md § "Quarterly restore drill".
|
|
# Hits the same backup function with ?mode=drill_reminder, which short-circuits
|
|
# the pipeline and only posts to Discord.
|
|
drill_scheduler = cloudscheduler.Job(
|
|
"dr-drill-scheduler",
|
|
name="dr-restore-drill-reminder",
|
|
description="Quarterly nudge to run the DR restore drill",
|
|
region=REGION,
|
|
# 15:00 UTC on the 1st of Jan/Apr/Jul/Oct
|
|
schedule="0 15 1 1,4,7,10 *",
|
|
time_zone="UTC",
|
|
http_target=cloudscheduler.JobHttpTargetArgs(
|
|
uri=function.url.apply(lambda u: f"{u}?mode=drill_reminder"),
|
|
http_method="POST",
|
|
oidc_token=cloudscheduler.JobHttpTargetOidcTokenArgs(
|
|
service_account_email=sa.email,
|
|
# OIDC audience must match the function URL without the query string
|
|
audience=function.url,
|
|
),
|
|
),
|
|
retry_config=cloudscheduler.JobRetryConfigArgs(
|
|
retry_count=1,
|
|
min_backoff_duration="60s",
|
|
max_backoff_duration="300s",
|
|
),
|
|
)
|
|
resources["drill_scheduler"] = drill_scheduler
|
|
|
|
return resources
|