Files
Andrew Beveridge ☄️andClaude Opus 4.8 6fa4400d3d feat(dr): back up all GitHub repos to S3 (survive a GitHub ban) (#929)
## Summary

Extends the nightly DR backup Cloud Function (`backup-to-aws`) to bundle
**every git repo under `github.com/nomadkaraoke` and
`github.com/beveradb`** into a per-repo `git bundle` on S3, so code +
full history survive a complete loss of GitHub access (e.g. an automated
copyright-strike account ban). Each bundle is directly cloneable —
restore is `git clone --mirror <bundle>` → `git push --mirror
<new-remote>`.

## Changes

- **`git_repos_export.py`** (new): enumerate repos via GitHub
`/user/repos` (includes private), mirror-clone + `git bundle create
--all`, upload to `git-repos/{owner}/{repo}.bundle` in staging, plus a
`manifest.json` inventory. Forks excluded, oversized/empty repos
skipped, per-repo errors non-fatal, org backed up **before** the
personal account for deadline resilience. PAT is read from Secret
Manager at **runtime** (not a deploy-time secret env var) so the
function deploys before the secret has a value and skips gracefully if
absent.
- **`main.py`**: new nightly Step 4b (between secrets export and S3
upload).
- **`infrastructure/modules/backup.py`**: bump function to 4Gi/2CPU
(mirror-clone uses in-memory `/tmp`); add `GIT_BACKUP_OWNERS` env.
- **`infrastructure/modules/secrets.py`**: new `github-backup-token`
secret (value added manually).
- **`.github/workflows/dr-backup-freshness.yml`**: monitor `git-repos/`
at 36h, counting only `.bundle` objects.
- **`docs/DISASTER-RECOVERY.md`**: new "Scenario 3: Loss of GitHub
access" + restore steps + token setup.

## Security

- The GitHub PAT is injected via a `GIT_CONFIG_*` `Authorization` header
(like `actions/checkout`), never in the clone URL/argv — so it can't
leak via `ps`, `TimeoutExpired`, or captured stderr. Token is also
redacted from any error text persisted into `manifest.json`.

## Testing

- 14 new unit tests for the module
(enumerate/filter/bundle/skip/error/security paths); full
`backup_to_aws` suite: **27 passing**.
- CodeRabbit local review (cycle 1) completed; all 4 findings (1
critical PAT leak, 2 major, 1 minor) addressed. Cycle-2 verification was
rate-limited (free-tier), so **left for the GitHub bot to review the
final diff**.

## Deploy notes

- **Pulumi already applied locally** (`github-backup-token` secret
created + function on 4Gi/2CPU) — CI re-run is a no-op.
- After merge, run `infrastructure/functions/backup_to_aws/deploy.sh` to
ship the new function source, then set the PAT:
`printf %s '<PAT>' | gcloud secrets versions add github-backup-token
--data-file=- --project=nomadkaraoke`
- Requires a classic GitHub PAT (owner: beveradb, member of nomadkaraoke
org) with `repo` + `read:org`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---
@coderabbitai ignore

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-23 00:44:52 -04:00

289 lines
12 KiB
Python

"""
Backup infrastructure module.
Creates:
- GCS staging bucket for backup exports
- Service account for backup Cloud Function
- Cloud Function v2 (Gen2) for nightly backups
- Cloud Scheduler job for daily trigger
"""
import pulumi
import pulumi_gcp as gcp
from pulumi_gcp import (
cloudfunctionsv2,
cloudscheduler,
serviceaccount,
storage,
)
from config import PROJECT_ID, REGION, get_project_number
def create_backup_resources(all_secrets: dict) -> dict:
"""Create all backup infrastructure resources."""
resources = {}
# ==================== Service Account ====================
sa = serviceaccount.Account(
"backup-to-aws-sa",
account_id="backup-to-aws",
display_name="Backup to AWS Function",
description="Service account for nightly backup Cloud Function",
)
resources["service_account"] = sa
# Firestore export permission
resources["firestore_export"] = gcp.projects.IAMMember(
"backup-firestore-export",
project=PROJECT_ID,
role="roles/datastore.importExportAdmin",
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
# BigQuery read access
resources["bq_viewer"] = gcp.projects.IAMMember(
"backup-bigquery-viewer",
project=PROJECT_ID,
role="roles/bigquery.dataViewer",
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
resources["bq_job_user"] = gcp.projects.IAMMember(
"backup-bigquery-job-user",
project=PROJECT_ID,
role="roles/bigquery.jobUser",
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Secret Manager access (for AWS credentials + nightly secrets backup)
resources["secret_accessor"] = gcp.projects.IAMMember(
"backup-secret-accessor",
project=PROJECT_ID,
role="roles/secretmanager.secretAccessor",
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
# secretmanager.viewer is required to LIST secrets — accessor only allows reading
# a known name. Backup needs to enumerate every secret in the project.
resources["secret_viewer"] = gcp.projects.IAMMember(
"backup-secret-viewer",
project=PROJECT_ID,
role="roles/secretmanager.viewer",
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Logging
resources["logging"] = gcp.projects.IAMMember(
"backup-logging",
project=PROJECT_ID,
role="roles/logging.logWriter",
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
# ==================== Staging Bucket ====================
staging_bucket = storage.Bucket(
"backup-staging-bucket",
name="nomadkaraoke-backup-staging",
location="US-CENTRAL1",
force_destroy=True, # Staging only — OK to destroy
uniform_bucket_level_access=True,
lifecycle_rules=[
storage.BucketLifecycleRuleArgs(
action=storage.BucketLifecycleRuleActionArgs(type="Delete"),
condition=storage.BucketLifecycleRuleConditionArgs(age=7),
),
],
)
resources["staging_bucket"] = staging_bucket
# SA needs objectAdmin on staging bucket (read + write + delete)
resources["staging_bucket_access"] = storage.BucketIAMMember(
"backup-staging-bucket-access",
bucket=staging_bucket.name,
role="roles/storage.objectAdmin",
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Firestore service agent needs write access to staging bucket for exports
resources["firestore_agent_staging_access"] = storage.BucketIAMMember(
"backup-firestore-agent-staging-access",
bucket=staging_bucket.name,
role="roles/storage.objectAdmin",
member=f"serviceAccount:service-{get_project_number()}@gcp-sa-firestore.iam.gserviceaccount.com",
)
# SA needs objectViewer on source buckets
source_buckets = [
"karaoke-gen-storage-nomadkaraoke",
"nomadkaraoke-kn-data",
]
for i, bucket_name in enumerate(source_buckets):
resources[f"source_bucket_access_{i}"] = storage.BucketIAMMember(
f"backup-source-bucket-access-{i}",
bucket=bucket_name,
role="roles/storage.objectViewer",
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
# ==================== Function Source Bucket ====================
function_source_bucket = storage.Bucket(
"backup-function-source",
name=f"backup-to-aws-source-{PROJECT_ID}",
location="US-CENTRAL1",
force_destroy=True,
uniform_bucket_level_access=True,
)
resources["function_source_bucket"] = function_source_bucket
# ==================== Cloud Function ====================
function = cloudfunctionsv2.Function(
"backup-to-aws-function",
name="backup-to-aws",
location=REGION,
description="Nightly backup of Firestore, BigQuery, and GCS to AWS S3",
build_config=cloudfunctionsv2.FunctionBuildConfigArgs(
runtime="python312",
entry_point="backup_to_aws",
source=cloudfunctionsv2.FunctionBuildConfigSourceArgs(
storage_source=cloudfunctionsv2.FunctionBuildConfigSourceStorageSourceArgs(
bucket=function_source_bucket.name,
object="backup-to-aws-source.zip",
),
),
),
service_config=cloudfunctionsv2.FunctionServiceConfigArgs(
# 4Gi/2CPU (up from 2Gi/1CPU): the git-repo backup mirror-clones each
# repo into the in-memory /tmp (Cloud Run tmpfs counts against
# memory) before bundling. Repos are processed serially and cleaned
# up between each, so peak is one repo's working set, but the bump
# gives headroom for the largest repo + faster clone/pack on 2 CPU.
available_memory="4Gi",
available_cpu="2",
# Kept in lockstep with the scheduler's attempt_deadline (1800s, the
# Cloud Scheduler max) below. If the function could run longer than
# the scheduler waits, runs in that gap would re-trigger the very
# timeout→retry→"Path already exists" collision this aligns to avoid.
timeout_seconds=1800,
min_instance_count=0,
max_instance_count=1,
service_account_email=sa.email,
environment_variables={
"GCP_PROJECT": PROJECT_ID,
"STAGING_BUCKET": "nomadkaraoke-backup-staging",
"S3_BUCKET": "nomadkaraoke-backup",
# Curve25519 public key (hex) for sealed-box encryption of secrets backup.
# The matching private key lives only in 1Password — function cannot decrypt.
# Set via: pulumi config set backupEncryptionPubkey <64-char-hex>
"BACKUP_ENCRYPTION_PUBKEY": pulumi.Config().require("backupEncryptionPubkey"),
# GitHub owners whose repos get bundled to S3 nightly (comma-sep).
# Backs up code + full history so it survives a GitHub account ban.
"GIT_BACKUP_OWNERS": "nomadkaraoke,beveradb",
},
secret_environment_variables=[
cloudfunctionsv2.FunctionServiceConfigSecretEnvironmentVariableArgs(
key="DISCORD_WEBHOOK_URL",
project_id=PROJECT_ID,
secret=all_secrets["discord-alert-webhook"].secret_id,
version="latest",
),
# NOTE: the GitHub backup PAT (secret "github-backup-token") is
# deliberately NOT wired as a secret env var. A version=latest
# env var fails the function deploy until the secret has a value,
# but the secret is created empty (value added manually). Instead
# the function reads it from Secret Manager at *runtime*
# (git_repos_export.get_github_token) and skips the git-repo step
# if it's absent — so deploy never blocks on it. The SA already
# holds project-wide secretmanager.secretAccessor.
],
),
)
resources["function"] = function
# Allow Cloud Scheduler to invoke the function
resources["scheduler_invoker"] = cloudfunctionsv2.FunctionIamMember(
"backup-scheduler-invoker",
project=PROJECT_ID,
location=REGION,
cloud_function=function.name,
role="roles/cloudfunctions.invoker",
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
# Gen2 functions need Cloud Run invoker too
resources["run_invoker"] = gcp.cloudrunv2.ServiceIamMember(
"backup-sa-run-invoker",
project=PROJECT_ID,
location=REGION,
name=function.name,
role="roles/run.invoker",
member=sa.email.apply(lambda email: f"serviceAccount:{email}"),
)
# ==================== Cloud Scheduler ====================
scheduler = cloudscheduler.Job(
"backup-scheduler",
name="backup-to-aws-nightly",
description="Trigger nightly backup to AWS S3",
region=REGION,
schedule="0 1 * * *",
time_zone="America/New_York",
# The backup pipeline (esp. the Firestore export) runs for several
# minutes. Without an explicit deadline, Cloud Scheduler defaults to
# 180s, times out the HTTP call, and retries — while the original
# invocation keeps running to success. The retry then collides on the
# already-created firestore/<date>/ export path ("400 Path already
# exists"), producing a spurious FAILED backup report every night.
# 1800s (30m, the Scheduler max) lets it wait for the real result.
attempt_deadline="1800s",
http_target=cloudscheduler.JobHttpTargetArgs(
uri=function.url,
http_method="POST",
oidc_token=cloudscheduler.JobHttpTargetOidcTokenArgs(
service_account_email=sa.email,
),
),
retry_config=cloudscheduler.JobRetryConfigArgs(
retry_count=2,
min_backoff_duration="300s",
max_backoff_duration="1800s",
),
)
resources["scheduler"] = scheduler
# Quarterly DR restore drill reminder — posts a Discord nudge so the human
# remembers to run the drill in docs/DISASTER-RECOVERY.md § "Quarterly restore drill".
# Hits the same backup function with ?mode=drill_reminder, which short-circuits
# the pipeline and only posts to Discord.
drill_scheduler = cloudscheduler.Job(
"dr-drill-scheduler",
name="dr-restore-drill-reminder",
description="Quarterly nudge to run the DR restore drill",
region=REGION,
# 15:00 UTC on the 1st of Jan/Apr/Jul/Oct
schedule="0 15 1 1,4,7,10 *",
time_zone="UTC",
http_target=cloudscheduler.JobHttpTargetArgs(
uri=function.url.apply(lambda u: f"{u}?mode=drill_reminder"),
http_method="POST",
oidc_token=cloudscheduler.JobHttpTargetOidcTokenArgs(
service_account_email=sa.email,
# OIDC audience must match the function URL without the query string
audience=function.url,
),
),
retry_config=cloudscheduler.JobRetryConfigArgs(
retry_count=1,
min_backoff_duration="60s",
max_backoff_duration="300s",
),
)
resources["drill_scheduler"] = drill_scheduler
return resources