Files
karaoke-gen/docs/DISASTER-RECOVERY.md
Andrew Beveridge ☄️andClaude Opus 5.5 a6d4163d2e feat(alerts): route non-signal Discord posts to #noise (v0.278.8) (#1186)
## Summary
Andrew asked for #alerts to carry only real problems. Routine and
informational posts now go to the new **#noise** channel. The GitHub org
PR webhook was already moved to #noise separately.

| Message | Before | After |
|---|---|---|
| Error monitor: New Error Pattern / Spike / Incident | #alerts |
#alerts |
| Error monitor: ✅ Patterns Auto-Resolved | #alerts | **#noise** |
| Error monitor: 📊 Daily Error Digest | #alerts | **#noise** |
| Nightly Backup Report (clean run) | #alerts | **#noise** |
| Nightly backup failed/partial (title is now "🚨 Nightly Backup FAILED")
| #alerts | #alerts |
| ✅ E2E Daily Test PASSED | #alerts | **#noise** |
| 🩹 E2E self-healed Stripe (auto-merge PR opened) | #alerts | **#noise**
|
| 🩹 self-heal PR needs CI approval click / could not open | #alerts |
#alerts |
| E2E / Tenant E2E failures, DR freshness stale, Job failed, Publish
completeness SHADOW, Gemini unavailable, OAuth credential alerts,
quarterly DR drill reminder | #alerts | #alerts (unchanged) |

## How the noise webhook is resolved
- **Backend / error monitor and the backup function** read it at
runtime: env `DISCORD_NOISE_WEBHOOK_URL` first, then the Secret Manager
secret `discord-noise-webhook` (the name can be overridden with
`DISCORD_NOISE_WEBHOOK_SECRET_NAME`). No Pulumi change is needed,
because both service accounts already have project-wide
`secretAccessor`.
- **GitHub Actions** uses `secrets.DISCORD_NOISE_WEBHOOK_URL ||
secrets.DISCORD_ALERT_WEBHOOK_URL`.
- **Nothing is silently dropped.** If the noise webhook can't be
resolved, or a post to it fails, the message goes to #alerts instead. In
the workflow this uses `curl -fsS` with a retry on the alert webhook.
- The error monitor now rate-limits each channel separately (10 per run
each), so noise can't use up the alert budget during an incident.

## Deploy notes
- **Backup Cloud Function:** CI doesn't deploy its source. Run
`infrastructure/functions/backup_to_aws/deploy.sh` after merge.
- **Error monitor job:** runs `karaoke-backend:latest`, so it picks up
the change once CI pushes the image.
- **No `pulumi up` needed.**

## Testing
- `tests/unit/services/error_monitor`: 376 passed. New tests cover noise
vs alert routing, fallback when the noise webhook is missing or fails,
separate rate limits, and secret resolution.
- `backend/tests` (excluding emulator): 6418 passed, 36 skipped.
- `infrastructure/functions/backup_to_aws`: 60 passed, including the new
`test_discord_alert.py` and routing tests in `test_main.py`. These were
run in a scratch venv because `functions_framework` isn't in the conda
env, which was already the case before this change.
- `actionlint` on `e2e-daily.yml`: no new findings. The only findings
are info-level shellcheck notes that were already there.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019pmG2XAgtUguEJWLFpkKHN

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 02:46:50 +00:00

29 KiB
Raw Permalink Blame History

Disaster Recovery Runbook

Last updated: 2026-10-08 Design doc: docs/archive/2026-03-27-business-continuity-design.md Implementation plan: docs/archive/2026-03-29-business-continuity-plan.md External services reference: docs/archive/2026-03-29-external-services-config.md

Quick Reference

Data Backup Location Recovery Method RPO
Firestore S3 firestore/YYYY-MM-DD/ Import to new GCP project 7 days off-site (S3); 24h local (GCS staging)
BigQuery (weekly) S3 bigquery/daily-refresh/ Load Parquet files 7 days
BigQuery (monthly) S3 bigquery/musicbrainz/ Load Parquet files 30 days
BigQuery (Spotify) S3 bigquery/spotify/ Load from Glacier Deep Archive N/A (static)
GCS job files GCS only — object versioning (non-current kept 7d since 2026-10-03) + soft-delete (7d). Off-site S3 copy stopped 2026-10-01; S3 gcs/job-files/ is a frozen snapshot up to 2026-09-30 Restore non-current/soft-deleted generation in GCS N/A off-site (frozen); in-GCS recovery window 7–30d
User-uploaded inputs (mix, own instrumental, lyrics, style uploads) jobs/{id}/input/** (KEEP forever; copied from uploads/{id}/ by every pipeline worker, backend/services/input_persistence.py). uploads/ expires after 180 days (was 7 until 2026-10-08) Read from jobs/{id}/input/; backfill stragglers with POST /api/admin/persist-uploads?dry_run=false Irreplaceable: no off-site copy. Lost for 64 Jan–Feb 2026 consumer jobs + BYO-instrumental jobs before the fix
Git repos (code) S3 git-repos/{owner}/{repo}.bundle (+ manifest.json) git clone <bundle> then push to a new remote 7 days (weekly, Sundays; incremental)
Secret Manager S3 secrets/YYYY-MM-DD.bin (sealed-box encrypted) Decrypt with private key from KeepassXC 24h
AWS credentials (function) GCP Secret Manager aws-backup-credentials Read directly On-change
AWS credentials (offline) KeepassXC — "Nomad Karaoke — DR backup AWS access" Manual lookup On-change
Backup decryption key KeepassXC — "Nomad Karaoke — DR backup decryption key" Manual lookup One-time

Prerequisites — verify these BEFORE you need them

  • KeepassXC contains "Nomad Karaoke — DR backup AWS access" (access key, secret key, region, S3 bucket name)
  • KeepassXC contains "Nomad Karaoke — DR backup decryption key" (Curve25519 private key, 64 hex chars)
  • GitHub repo nomadkaraoke/karaoke-gen has the secrets AWS_BACKUP_READONLY_ACCESS_KEY_ID, AWS_BACKUP_READONLY_SECRET_ACCESS_KEY, DR_MONITOR_DISCORD_WEBHOOK configured (powers the freshness monitor)
  • GCP Secret Manager github-backup-token holds a valid GitHub PAT (repo + read:org) so the weekly git step can bundle every repo — without it the git-repo backup silently skips (see § "Git repo backup setup")
  • You can decrypt yesterday's secrets backup locally (run the drill in § "Quarterly restore drill")
  • Your KeepassXC database itself is backed up off-machine (cloud sync, second device, or printed paper recovery)

If any of these fail, fix them now — not during an incident.

Scenario 1: Accidental Data Deletion (GCP Still Available)

Firestore Recovery

Option A — last night's local export (fastest, within 24h):

Point-in-time recovery was disabled on 2026-09-26 (GCP cost cut), so there is no PITR restore any more. The backup function exports Firestore nightly to gs://nomadkaraoke-backup-staging/firestore/<YYYY-MM-DD>/ (only the latest night is kept):

LATEST=$(gcloud storage ls gs://nomadkaraoke-backup-staging/firestore/ | sort | tail -1)
gcloud firestore import "$LATEST" --project=nomadkaraoke

Option B — Import from S3 backup:

# Download latest export from S3 (set creds first — see § "Setting AWS creds locally")
aws s3 sync s3://nomadkaraoke-backup/firestore/LATEST_DATE/ /tmp/firestore-restore/

# Upload to GCS
gsutil -m cp -r /tmp/firestore-restore/ gs://nomadkaraoke-backup-staging/firestore-import/

# Import
gcloud firestore import gs://nomadkaraoke-backup-staging/firestore-import/ --project=nomadkaraoke

BigQuery Recovery

Option A — Time Travel (within 7 days):

-- Query historical data
SELECT * FROM `nomadkaraoke.karaoke_decide.TABLE_NAME`
FOR SYSTEM_TIME AS OF TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 HOUR);

-- Restore to a new table
CREATE TABLE `nomadkaraoke.karaoke_decide.TABLE_NAME_restored` AS
SELECT * FROM `nomadkaraoke.karaoke_decide.TABLE_NAME`
FOR SYSTEM_TIME AS OF TIMESTAMP("2026-03-28 12:00:00 UTC");

Option B — Load from S3 backup:

# Download Parquet exports
aws s3 sync s3://nomadkaraoke-backup/bigquery/daily-refresh/LATEST_DATE/ /tmp/bq-restore/

# Load each table
bq load --source_format=PARQUET \
  nomadkaraoke:karaoke_decide.TABLE_NAME \
  /tmp/bq-restore/TABLE_NAME/*.parquet

GCS Recovery

The primary bucket has two independent recovery paths. Pick whichever applies.

Option A — GCS soft-delete (within 7 days, default-on, no special config):

# List soft-deleted objects under a prefix
gcloud storage ls --soft-deleted gs://karaoke-gen-storage-nomadkaraoke/jobs/JOB_ID/

# Restore a specific soft-deleted object (note the generation number from the listing)
gcloud storage restore gs://karaoke-gen-storage-nomadkaraoke/jobs/JOB_ID/file.flac#GENERATION

Option B — Object versioning (within 30 days, restores overwritten/deleted versions):

Requires versioning to be enabled on the bucket — confirm with gcloud storage buckets describe gs://karaoke-gen-storage-nomadkaraoke --format="value(versioning.enabled)". The Pulumi config in infrastructure/modules/storage.py enables it; if the live bucket reports False, run pulumi up to reconcile before relying on this path.

# List all versions (including non-current) of a path
gcloud storage ls -a gs://karaoke-gen-storage-nomadkaraoke/jobs/JOB_ID/

# Copy a specific non-current generation back to live
gcloud storage cp gs://karaoke-gen-storage-nomadkaraoke/jobs/JOB_ID/file.flac#GENERATION \
  gs://karaoke-gen-storage-nomadkaraoke/jobs/JOB_ID/file.flac

Option C — Restore from S3 (if A and B both miss the window; only works for files that existed by 2026-09-30 — the nightly S3 copy of job files was stopped 2026-10-01, see § "Job files: off-site copy stopped"):

aws s3 sync s3://nomadkaraoke-backup/gcs/job-files/jobs/JOB_ID/ /tmp/gcs-restore/

gsutil -m cp -r /tmp/gcs-restore/ gs://karaoke-gen-storage-nomadkaraoke/jobs/JOB_ID/

Secret Manager Recovery

Option A — GCP Secret Manager versioning (Secret Manager keeps every version by default):

# List versions
gcloud secrets versions list SECRET_NAME --project=nomadkaraoke

# Inspect any prior version
gcloud secrets versions access VERSION_NUMBER --secret=SECRET_NAME --project=nomadkaraoke

# Restore: create a new "latest" version with the old payload
gcloud secrets versions access VERSION_NUMBER --secret=SECRET_NAME --project=nomadkaraoke \
  | gcloud secrets versions add SECRET_NAME --data-file=- --project=nomadkaraoke

Option B — Decrypt the encrypted S3 backup (use if a secret was deleted entirely): See § "Decrypting the secrets backup" below.

Scenario 2: Complete GCP Project Loss

Prerequisites

  • AWS credentials from KeepassXC ("Nomad Karaoke — DR backup AWS access")
  • Backup decryption key from KeepassXC ("Nomad Karaoke — DR backup decryption key")
  • All GitHub repos (code + IaC)
  • A working machine with Python 3.10+ and pynacl installable

Step-by-Step Recovery

1. Create new GCP project

gcloud projects create nomadkaraoke-v2 --name="Nomad Karaoke"
gcloud config set project nomadkaraoke-v2
gcloud services enable firestore.googleapis.com run.googleapis.com \
  cloudfunctionsv2.googleapis.com cloudscheduler.googleapis.com \
  secretmanager.googleapis.com bigquery.googleapis.com \
  cloudtasks.googleapis.com artifactregistry.googleapis.com

2. Set AWS credentials locally — see § "Setting AWS creds locally" below.

3. Restore Firestore

# Find the latest backup date
LATEST_DATE=$(aws s3 ls s3://nomadkaraoke-backup/firestore/ | awk '{print $2}' | tr -d '/' | sort | tail -1)
echo "Restoring from $LATEST_DATE"

# Create a temporary GCS bucket in the new project to land the import
gsutil mb -l US-CENTRAL1 gs://nomadkaraoke-v2-firestore-import/

aws s3 sync "s3://nomadkaraoke-backup/firestore/${LATEST_DATE}/" /tmp/firestore-restore/
gsutil -m cp -r /tmp/firestore-restore/ gs://nomadkaraoke-v2-firestore-import/

# The Firestore import requires the metadata file emitted by the export — find it
META=$(gsutil ls "gs://nomadkaraoke-v2-firestore-import/firestore-restore/**/all_namespaces*.overall_export_metadata" | head -1)
gcloud firestore import "$META" --project=nomadkaraoke-v2

4. Restore BigQuery

# Create dataset
bq mk --dataset nomadkaraoke-v2:karaoke_decide

# Find latest dates
DAILY_DATE=$(aws s3 ls s3://nomadkaraoke-backup/bigquery/daily-refresh/ | awk '{print $2}' | tr -d '/' | sort | tail -1)
MONTHLY_DATE=$(aws s3 ls s3://nomadkaraoke-backup/bigquery/musicbrainz/ | awk '{print $2}' | tr -d '/' | sort | tail -1)

for spec in "daily-refresh|$DAILY_DATE" "musicbrainz|$MONTHLY_DATE"; do
  prefix=${spec%|*}
  date=${spec#*|}
  aws s3 sync "s3://nomadkaraoke-backup/bigquery/$prefix/$date/" "/tmp/bq-$prefix/"
  for dir in /tmp/bq-$prefix/*/; do
    table=$(basename "$dir")
    bq load --source_format=PARQUET nomadkaraoke-v2:karaoke_decide.$table "$dir*.parquet"
  done
done

Spotify tables (Glacier Deep Archive): Retrieval takes 12-48 hours.

# Initiate restore (bulk = cheapest)
aws s3api restore-object --bucket nomadkaraoke-backup \
  --key "bigquery/spotify/TABLE/part-000.parquet" \
  --restore-request '{"Days":7,"GlacierJobParameters":{"Tier":"Bulk"}}'
# Wait 12-48 hours, then download and load

5. Restore GCS files

# Create new bucket
gsutil mb -l US-CENTRAL1 gs://karaoke-gen-storage-nomadkaraoke-v2/

# Sync from S3 (146 GB last we measured — needs a host with comparable disk + bandwidth).
# NOTE: frozen snapshot — job files created/changed after 2026-09-30 are NOT in S3
# (off-site copy stopped 2026-10-01). Newer outputs must be regenerated.
aws s3 sync s3://nomadkaraoke-backup/gcs/job-files/ /tmp/gcs-restore/
gsutil -m cp -r /tmp/gcs-restore/ gs://karaoke-gen-storage-nomadkaraoke-v2/

6. Restore secrets — decrypt the latest sealed-box backup and recreate each secret.

# Get the latest encrypted bundle
LATEST=$(aws s3 ls s3://nomadkaraoke-backup/secrets/ | awk '{print $NF}' | sort | tail -1)
aws s3 cp "s3://nomadkaraoke-backup/secrets/$LATEST" /tmp/secrets.bin

# Install decryption tool if needed
pip install pynacl

# Decrypt — pulls private key from KeepassXC
PRIVATE_KEY_HEX="<paste from KeepassXC 'DR backup decryption key'>"
python3 - <<PY > /tmp/secrets.json
from nacl.public import PrivateKey, SealedBox
import os, json
sk = PrivateKey(bytes.fromhex(os.environ['PRIVATE_KEY_HEX']))
ct = open('/tmp/secrets.bin', 'rb').read()
print(SealedBox(sk).decrypt(ct).decode('utf-8'))
PY

# Recreate every secret in the new project. Review the generated commands first.
python3 - <<'PY' > /tmp/restore_secrets.sh
import json, base64, shlex
bundle = json.load(open('/tmp/secrets.json'))
for name, entry in bundle['secrets'].items():
    if 'error' in entry:
        print(f"# SKIPPED {name}: {entry['error']}", )
        continue
    raw = entry['value']
    if entry['encoding'] == 'base64':
        # Pipe binary through base64 -d to avoid shell-escaping issues
        print(f"echo {shlex.quote(raw)} | base64 -d | gcloud secrets create {name} --data-file=- --project=nomadkaraoke-v2")
    else:
        print(f"printf %s {shlex.quote(raw)} | gcloud secrets create {name} --data-file=- --project=nomadkaraoke-v2")
PY

# Inspect, then execute
less /tmp/restore_secrets.sh
bash /tmp/restore_secrets.sh

# Cleanup
shred -u /tmp/secrets.json /tmp/secrets.bin /tmp/restore_secrets.sh
unset PRIVATE_KEY_HEX

7. Set Pulumi config for new project

cd karaoke-gen/infrastructure
pulumi stack init disaster-recovery --copy-config-from prod
pulumi config set gcp:project nomadkaraoke-v2
# The backup encryption pubkey from the prior project still works — its private
# key is in KeepassXC. Either copy it from the old stack or set fresh.
pulumi config set backupEncryptionPubkey <hex>

8. Rebuild infrastructure

pulumi up

9. Update CI/CD to target the new project

GitHub Actions secrets reference nomadkaraoke (old project ID). Update them so deploys land in the new project:

# In the karaoke-gen repo, update these secrets to the new project / new SA key:
gh secret set GCP_PROJECT_ID --body "nomadkaraoke-v2"
gh secret set GCP_SA_KEY --body "$(cat ~/path/to/new-deploy-sa-key.json)"
# (If using Workload Identity Federation, update the audience instead.)

Then trigger a deploy or git push to redeploy backend + frontend.

10. Update DNS (Cloudflare)

Update DNS records to point to new Cloud Run URLs. See docs/archive/2026-03-29-external-services-config.md.

11. Re-register OAuth apps and rotate API keys

Some external services lock to redirect URIs or treat the old GCP project ID as part of their config. Walk through docs/archive/2026-03-29-external-services-config.md end-to-end:

  • Spotify Web API + Spotify Connect (redirect URIs)
  • Google OAuth (redirect URIs, OAuth consent screen)
  • Dropbox app (redirect URIs)
  • YouTube OAuth (redirect URIs)
  • Stripe webhook endpoint URL (secret regenerates)
  • AudioShake API key (rotate as a precaution if old project was compromised)
  • Gemini Developer API key (gemini-api-key secret; key lives in an AI Studio project outside nomadkaraoke — just copy the secret)
  • Discord webhook URLs (re-issue if old GCP exposure could have leaked them)
  • Cloudflare API tokens (rotate)

Scenario 3: Loss of GitHub access (account ban / repo takedown)

If the GitHub account is suspended (e.g. an automated copyright false-positive) or repos are taken down, the code and its full history are still in S3. The weekly backup bundles every repo under github.com/nomadkaraoke and github.com/beveradb into a single git bundle per repo — a self-contained file you can clone from directly, no GitHub required. GCP is unaffected, so production keeps running; this is purely about recovering the source of truth to a new home.

What's backed up

  • s3://nomadkaraoke-backup/git-repos/{owner}/{repo}.bundle — one bundle per repo, containing all branches and tags (entire reachable history).
  • s3://nomadkaraoke-backup/git-repos/manifest.json — inventory of every repo at backup time: visibility, default branch, description, size, fork/archived flags, and per-repo backup status. Use it to know exactly what existed and to recreate repo settings on the new host.

Forks are excluded by default (recoverable from upstream). The git step runs weekly (Sundays) and is incremental (since 2026-09-26, to cut cross-cloud egress): a repo is only re-cloned and re-uploaded when its GitHub pushed_at is newer than its bundle's S3 upload time (minus a 1h safety lag); unchanged repos show "status": "unchanged" in the manifest and keep their existing S3 bundle. git-repos/ only uploads to S3 in the same invocation as the git step (a bundle stranded in staging by a failed upload is re-bundled next Sunday rather than shipped late, which would break the lag assumption). The first Sunday of each month re-bundles everything. So a bundle is at most ~7 days behind (7-day RPO). To force a run now: ?mode=git_repos (incremental) or ?mode=git_repos&full=1 (all repos) — see § "Git repo backup setup".

Restore

# 1. Set AWS creds (see § "Setting AWS creds locally") and pull the bundles.
aws s3 sync s3://nomadkaraoke-backup/git-repos/ /tmp/git-repos/

# 2. Review the inventory — what existed, and did every repo back up cleanly?
jq '{total, bundled, errors, skipped, repos: [.repos[] | {full_name, status, default_branch, private}]}' \
  /tmp/git-repos/manifest.json

# 3. Reconstruct the repo from any bundle. Use --mirror so EVERY branch and tag
#    is restored as a real ref — a plain `git clone` keeps non-default branches
#    only as remote-tracking refs, and `git push --all` would then omit them.
cd /tmp/git-repos/nomadkaraoke
git clone --mirror karaoke-gen.bundle karaoke-gen.git
cd karaoke-gen.git
git show-ref            # every branch + tag is present as a real ref

# 4. Point it at a new remote and push everything (--mirror pushes all refs).
#    New home options: a fresh GitHub org/account, GitLab, Codeberg, or a
#    self-hosted Gitea. Create the empty repo there first, then:
git remote set-url origin <NEW_REMOTE_URL>   # bare mirror already has 'origin'
git push --mirror origin
# (For a normal working clone afterwards: `git clone <NEW_REMOTE_URL>`.)

To restore all repos in one pass:

cd /tmp/git-repos
for owner in */; do
  for bundle in "$owner"*.bundle; do
    name=$(basename "$bundle" .bundle)
    # --mirror preserves all branches/tags; `git clone` creates leading dirs.
    git clone --mirror "$bundle" "restored/${owner}${name}.git"
    # then: cd in, `git remote set-url origin <NEW_URL>`, `git push --mirror origin`
  done
done

Verify a bundle without cloning: git bundle verify /tmp/git-repos/nomadkaraoke/karaoke-gen.bundle.

After restore — repoint tooling

The DR runbook, CI, poetry/pip VCS deps, and cross-repo scripts reference github.com/nomadkaraoke/... URLs. Once repos live at a new host, update:

  • Git remotes in every local clone / worktree
  • CI/CD config that clones sibling repos (e.g. flacfetch in karaoke-gen)
  • Any poetry/pip dependency pinned to a GitHub URL
  • Webhooks / deploy keys / Actions secrets on the new host
  • GIT_BACKUP_OWNERS + the github-backup-token PAT so the nightly backup keeps bundling from the new location

Git repo backup setup

One-time setup so the nightly job can bundle private repos:

  1. Create a GitHub PAT owned by beveradb (a member of the nomadkaraoke org):
    • Classic PAT: scopes repo (full, to read private repos) + read:org (to enumerate org repos). Simplest.
    • Fine-grained PAT: resource owner = beveradb and a second token for the nomadkaraoke org (fine-grained tokens are single-owner); grant Contents: Read-only + Metadata: Read-only. Classic is easier here.
  2. Store it in Secret Manager (value added manually — Pulumi only creates the empty secret resource):
    printf %s '<PAT>' | gcloud secrets versions add github-backup-token \
      --data-file=- --project=nomadkaraoke
    
    Also keep a copy in KeepassXC as "Nomad Karaoke — DR GitHub backup PAT".
  3. The owners backed up are set by the GIT_BACKUP_OWNERS env var on the backup-to-aws function (default nomadkaraoke,beveradb), wired in infrastructure/modules/backup.py.
  4. Trigger a git-only run (skips Firestore/BigQuery/GCS/secrets, so it is safe to run any time) and confirm bundles appear:
    curl -X POST -H "Authorization: Bearer $(gcloud auth print-identity-token)" \
      "https://us-central1-nomadkaraoke.cloudfunctions.net/backup-to-aws?mode=git_repos"
    aws s3 ls s3://nomadkaraoke-backup/git-repos/ --recursive | head
    

If the PAT is missing/expired the git-repo step logs a warning and skips; the freshness monitor then flags git-repos/manifest.json as stale within ~8 days.

Decrypting the secrets backup

# Set AWS creds (see § "Setting AWS creds locally")
LATEST=$(aws s3 ls s3://nomadkaraoke-backup/secrets/ | awk '{print $NF}' | sort | tail -1)
aws s3 cp "s3://nomadkaraoke-backup/secrets/$LATEST" /tmp/secrets.bin

pip install pynacl
PRIVATE_KEY_HEX="<paste from KeepassXC>"
python3 -c "
import os, json
from nacl.public import PrivateKey, SealedBox
sk = PrivateKey(bytes.fromhex(os.environ['PRIVATE_KEY_HEX']))
bundle = json.loads(SealedBox(sk).decrypt(open('/tmp/secrets.bin','rb').read()))
print(json.dumps({k: '***' for k in bundle['secrets']}, indent=2))
print('Exported at:', bundle['exported_at'])
"
shred -u /tmp/secrets.bin
unset PRIVATE_KEY_HEX

Setting AWS creds locally

The function reads creds from GCP Secret Manager. For local recovery work, retrieve them from KeepassXC (preferred — survives GCP outage) or from Secret Manager (if GCP is up):

# If GCP is still up:
CREDS=$(gcloud secrets versions access latest --secret=aws-backup-credentials --project=nomadkaraoke)
export AWS_ACCESS_KEY_ID=$(echo "$CREDS" | jq -r .access_key_id)
export AWS_SECRET_ACCESS_KEY=$(echo "$CREDS" | jq -r .secret_access_key)
export AWS_DEFAULT_REGION=$(echo "$CREDS" | jq -r .region)

# If GCP is gone — paste from KeepassXC:
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_DEFAULT_REGION=us-east-1

Backup freshness monitor

A GitHub Actions cron (.github/workflows/dr-backup-freshness.yml) runs daily at 14:00 UTC and checks that the most recent objects in s3://nomadkaraoke-backup/firestore/, secrets/, bigquery/daily-refresh/, and git-repos/manifest.json are no older than their per-prefix limit. Stale or missing backups → Discord alert + workflow failure.

Per-prefix limits reflect each prefix's S3 (off-site) cadence, which is not the same as its GCS-staging cadence:

Prefix S3 upload cadence Monitor limit
secrets/ nightly 36h
firestore/, bigquery/daily-refresh/ weekly (Sundays) 192h (≈8 days)
git-repos/manifest.json weekly (Sundays), incremental 192h (≈8 days) + manifest must show 0 bundle errors

Firestore exports to GCS staging nightly (24h local restore point) but only ships to S3 weekly to cut cross-cloud egress (see backup_to_aws/main.py → firestore_to_s3_today). The monitor only sees the S3 copy, so its Firestore limit must allow a full week — using the 36h nightly figure caused a false "DR backup is stale" alert every Tue–Sat (fixed 2026-06-18).

This monitor runs in GitHub, not GCP, so it survives a complete GCP project loss.

Required GitHub secrets (set via gh secret set):

  • AWS_BACKUP_READONLY_ACCESS_KEY_ID — IAM user with s3:ListBucket + s3:GetObject only on nomadkaraoke-backup
  • AWS_BACKUP_READONLY_SECRET_ACCESS_KEY
  • DR_MONITOR_DISCORD_WEBHOOK — Discord webhook URL (this monitor only posts on stale/failed, so #alerts is right). The nightly function itself posts a clean run's "Nightly Backup Report" to #noise (secret discord-noise-webhook, read at runtime) and a failed/partial run as "🚨 Nightly Backup FAILED" to #alerts — see ARCHITECTURE.md § Discord channels

Manual setup (one-time, in AWS Console):

  1. Create IAM user nomadkaraoke-backup-monitor (no console access, programmatic only)
  2. Attach inline policy granting s3:ListBucket + s3:GetObject on arn:aws:s3:::nomadkaraoke-backup and arn:aws:s3:::nomadkaraoke-backup/*
  3. Create access key pair, store in KeepassXC as "Nomad Karaoke — DR backup readonly access"
  4. gh secret set AWS_BACKUP_READONLY_ACCESS_KEY_ID --body '<key>' --repo nomadkaraoke/karaoke-gen
  5. gh secret set AWS_BACKUP_READONLY_SECRET_ACCESS_KEY --body '<secret>' --repo nomadkaraoke/karaoke-gen
  6. gh secret set DR_MONITOR_DISCORD_WEBHOOK --body '<url>' --repo nomadkaraoke/karaoke-gen
  7. Trigger a test run: gh workflow run dr-backup-freshness.yml --repo nomadkaraoke/karaoke-gen

Quarterly restore drill

Once per quarter, prove the chain works without committing to a real restore. Scheduler dr-restore-drill-reminder posts a Discord nudge each quarter; when it fires, do this:

# 1. Decrypt yesterday's secrets backup (proves backup + private key + tooling all work)
#    Follow § "Decrypting the secrets backup" above. Do NOT write secrets to disk in plaintext;
#    just confirm the count of decrypted secrets matches gcloud secrets list.
LIVE_COUNT=$(gcloud secrets list --project=nomadkaraoke --format=json | jq length)
echo "Live secret count: $LIVE_COUNT"
# Compare against the count printed by the decryption step.

# 2. Restore Firestore to a side database (does not touch production)
LATEST_DATE=$(aws s3 ls s3://nomadkaraoke-backup/firestore/ | awk '{print $2}' | tr -d '/' | sort | tail -1)
gsutil -m cp -r "s3://nomadkaraoke-backup/firestore/$LATEST_DATE/" gs://nomadkaraoke-backup-staging/drill-import/
META=$(gsutil ls "gs://nomadkaraoke-backup-staging/drill-import/$LATEST_DATE/**/all_namespaces*.overall_export_metadata" | head -1)
gcloud firestore databases create --database=drill-$(date +%Y%m%d) \
  --location=us-central1 --type=firestore-native --project=nomadkaraoke
gcloud firestore import "$META" --database=drill-$(date +%Y%m%d) --project=nomadkaraoke

# 3. Spot-check a few collections in the side database, then delete it
gcloud firestore databases delete drill-$(date +%Y%m%d) --project=nomadkaraoke

# 4. Restore one specific BigQuery table from S3 to a drill_ prefix
bq mk --dataset --description="restore drill" nomadkaraoke:drill_$(date +%Y%m%d)
DAILY_DATE=$(aws s3 ls s3://nomadkaraoke-backup/bigquery/daily-refresh/ | awk '{print $2}' | tr -d '/' | sort | tail -1)
aws s3 cp "s3://nomadkaraoke-backup/bigquery/daily-refresh/$DAILY_DATE/karaokenerds_raw/000000000000.parquet" /tmp/
bq load --source_format=PARQUET nomadkaraoke:drill_$(date +%Y%m%d).karaokenerds_raw /tmp/000000000000.parquet
bq query --use_legacy_sql=false "SELECT COUNT(*) FROM \`nomadkaraoke.drill_$(date +%Y%m%d).karaokenerds_raw\`"
bq rm -r -f -d nomadkaraoke:drill_$(date +%Y%m%d)

# 5. Note the date and result in docs/LESSONS-LEARNED.md

Datasets intentionally NOT in S3 (regenerable from upstream)

These datasets are large and publicly available, so backing them up to S3 would cost more than re-downloading them after a disaster. If you need them after a restore, regenerate from these sources:

Dataset Source Re-download notes
MusicBrainz raw dumps (~17 GB) https://musicbrainz.org/doc/MusicBrainz_Database/Download Daily replication available; full import is several hours via mbslave or Postgres dump
MLHD listening histories (~252 GB) https://ddmal.music.mcgill.ca/research/The_Music_Listening_Histories_Dataset/ Static dataset; download once, derived BigQuery tables are in monthly S3 backup

Only the derived BigQuery tables under bigquery/musicbrainz/ are in S3 (these contain joins/normalizations that would take hours to recompute even after raw re-import). The raw source files are not.

Cost Estimates for Recovery

Action Cost
S3 Glacier Deep Archive retrieval (1 TB, bulk) ~$150
S3 Glacier Deep Archive retrieval (1 TB, standard) ~$750
S3 data transfer out to GCP ~$0.09/GB
Full Spotify dataset retrieval + transfer (~100 GB) ~$25 total
Full GCS job-files egress (~150 GB) ~$14

Job files: off-site copy stopped (2026-10-01)

The nightly backup-to-aws step that delta-synced karaoke-gen-storage-nomadkaraoke (jobs/, tenants/, themes/) → staging gcs/job-files/ → S3 was removed on 2026-10-01 as a GCP cost cut (~$15/mo cross-cloud egress; business cash constraint). Job outputs are regenerable from their inputs, and the primary bucket keeps object versioning (non-current versions deleted 30 days after being superseded) plus 7-day soft-delete, which covers accidental overwrite/delete. What is not covered any more: loss of the whole GCP project/bucket for files created after 2026-09-30. The existing S3 gcs/job-files/ objects were deliberately left in place (frozen snapshot). Firestore (weekly), secrets (nightly), BigQuery, kn-data and git bundle backups are unchanged. The freshness monitor no longer checks gcs/job-files/.

To re-enable: restore the Step 3 sync_gcs_to_staging(source_bucket="karaoke-gen-storage-nomadkaraoke", staging_prefix="gcs/job-files/") call in infrastructure/functions/backup_to_aws/main.py (see git history) and re-add the gcs/job-files/ row to .github/workflows/dr-backup-freshness.yml.

Incident: S3 overwrites silently failing 2026-09-14 → 2026-09-26

The incremental S3 upload (#997) checked "already in S3?" with HeadObject, but the backup-writer IAM user only has s3:ListBucket + s3:PutObject (no s3:GetObject). S3 answers HEAD on a missing key with 404 (fine — the file uploaded) but on an existing key with 403, so every overwrite errored: git bundles + manifest.json stopped updating after 09-15 (the freshness monitor failed daily on git-repos/), and re-written job files (e.g. lyrics/corrections_updated.json, karaoke.lrc, re-rendered screens) kept their older S3 copy. New keys (dated secrets/Firestore exports, new job files) were unaffected. Fixed 2026-09-26 by switching the existence check to ListObjectsV2 (s3_upload._list_s3_sizes); a manual ?mode=git_repos run refreshed all changed bundles the same day. Re-written job files from 09-14..09-19 aged out of the 7-day staging lifecycle before the fix, so S3 holds their previous version (primary copies in GCS are intact; not backfilled). Lesson: the IAM user is intentionally write+list only — test new S3 calls against it, not an admin credential.