Files
karaoke-gen/scripts/storage_retention_report.py
T
Andrew Beveridge ☄️andClaude Opus 5.5 3ecea6f2e7 feat(storage): enable real purge with deletion audit log and orphan cleanup mode (v0.266.0) (#1127)
## Summary
Andrew approved this follow-up to #1125 and #1126. It turns on real
deletion, adds an audit log of every deleted object, and adds an
orphan-folder cleanup mode.

### Real deletion on
`STORAGE_RETENTION_DRY_RUN=false` and
`STORAGE_RETENTION_MAX_JOBS_PER_RUN=50` are added to the backend
deploy's `--set-env-vars`. The API service is the only thing that reads
them: it passes `--dry-run` to `storage-retention-job`. That keeps the
setting in one place, so a later deploy can't silently revert it.

### Deletion audit log
- `storage-retention/deletion-logs/<ts>-<mode>.jsonl` has one line per
deleted or failed object: timestamp, run_id, mode, job_id, path,
size_bytes, generation, category and result.
- The log is rewritten after every job or folder, so a crash still
leaves a complete record. Each run also writes
`<ts>-<mode>.summary.json`.
- Deletes are pinned to the logged generation (`if_generation_match`).
- `assert_deletable` refuses any path outside `jobs/{id}/`, so
`storage-retention/` is never touched. No lifecycle rule matches live
objects there.
- The Firestore manifest on each job doc now also records generations
and the log path.

### Orphan cleanup
- `POST /api/internal/storage-retention/orphans` runs on the same Cloud
Run job with `--mode orphans`.
- It targets `jobs/{id}/` folders with no doc in `jobs`, `jobs-dev` or
`youtube_upload_queue`, re-checked live per folder against a fresh
listing.
- It skips any folder with an object modified in the last 30 days, keeps
`input/`, and deletes the rest.
- `encoding_worker_jobs` doesn't count as a record: it's the encoder's
expiring result cache, which deleted jobs leave behind.
- It is a dry-run unless called with
`dry_run=false&confirm=delete-orphan-job-folders`, and the scheduler
never runs it.

### Docs
- `docs/STORAGE-RETENTION.md`: run modes, log locations, jq review
commands and restore one-liners (noncurrent `cp #gen` and soft-delete
`restore`).
- `scripts/storage_retention_report.py`: list runs, summarise a run, or
find which run deleted a path or job.

## Testing
New tests cover the log contents, generation-pinned deletes, per-job
flushing across a crash, the path guard, orphan planning (input kept,
30-day skip, live record re-check, batch limit), and the endpoint and
worker modes. Full backend suite: 5774 passed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added admin-triggered orphan-folder cleanup, with dry-run enabled by
default, optional folder limits, and explicit confirmation required for
deletion.
* Added deletion reports and audit logs covering removed objects,
reclaimed space, and failures, plus a read-only tool to review runs and
search records.
* **Bug Fixes**
* Added safeguards to prevent deleting changed objects or files outside
their intended job folders.
* **Documentation**
* Expanded storage-retention guidance with recovery details, review
commands, and cleanup instructions.
* **Chores**
* Set the production cleanup limit to 50 jobs per run and updated the
package version.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 11:54:05 -04:00

102 lines
3.8 KiB
Python

#!/usr/bin/env python3
"""
Review what the storage-retention process deleted (docs/STORAGE-RETENTION.md).
python scripts/storage_retention_report.py # list all runs (summaries)
python scripts/storage_retention_report.py RUN_ID # one run: totals by category + per job
python scripts/storage_retention_report.py --find jobs/abc/finals/x.mp4 # which run deleted it
python scripts/storage_retention_report.py --find abc123 # everything deleted for a job id
RUN_ID is the log name without extension, e.g. 20261004T103000Z-job_purge.
Uses Application Default Credentials (read-only access is enough).
"""
import argparse
import json
import sys
from collections import Counter, defaultdict
from google.cloud import storage
BUCKET = "karaoke-gen-storage-nomadkaraoke"
PREFIX = "storage-retention/deletion-logs/"
def _bucket():
return storage.Client(project="nomadkaraoke").bucket(BUCKET)
def _gib(n):
return f"{n / 2**30:.2f} GiB"
def list_runs(bucket):
for blob in sorted(bucket.list_blobs(prefix=PREFIX), key=lambda b: b.name):
if not blob.name.endswith(".summary.json"):
continue
s = json.loads(blob.download_as_text())
print(f"{s['run_id']:<40} jobs={s['jobs']:<5} objects={s['objects_deleted']:<6} "
f"{_gib(s['bytes_deleted']):>11} failures={s['failures']}")
def _lines(bucket, run_id):
blob = bucket.blob(f"{PREFIX}{run_id}.jsonl")
for line in blob.download_as_text().splitlines():
if line.strip():
yield json.loads(line)
def show_run(bucket, run_id):
by_cat, n_cat, by_job = Counter(), Counter(), defaultdict(Counter)
failures = []
for e in _lines(bucket, run_id):
if e["result"] == "deleted":
by_cat[e["category"]] += e["size_bytes"]
n_cat[e["category"]] += 1
by_job[e["job_id"]][e["category"]] += e["size_bytes"]
else:
failures.append(e)
print(f"Run {run_id}: {len(by_job)} jobs, {sum(n_cat.values())} objects, {_gib(sum(by_cat.values()))}")
for cat, size in by_cat.most_common():
print(f" {cat:<18} {n_cat[cat]:>6} objects {_gib(size):>11}")
print("Per job:")
for job_id, cats in sorted(by_job.items(), key=lambda kv: -sum(kv[1].values())):
print(f" {job_id:<16} {_gib(sum(cats.values())):>11} " + ", ".join(f"{c}={_gib(v)}" for c, v in cats.items()))
if failures:
print(f"Failures ({len(failures)}):")
for e in failures:
print(f" {e['path']}: {e.get('error')}")
def find(bucket, needle):
for blob in sorted(bucket.list_blobs(prefix=PREFIX), key=lambda b: b.name):
if not blob.name.endswith(".jsonl"):
continue
run_id = blob.name[len(PREFIX):-len(".jsonl")]
for e in _lines(bucket, run_id):
if needle == e["job_id"] or needle in e["path"]:
# Only offer a restore for objects actually deleted at a known
# generation (a failed delete may have a newer live version).
restore = ""
if e["result"] == "deleted" and e.get("generation"):
restore = (f" restore: gcloud storage cp --no-clobber 'gs://{BUCKET}/{e['path']}#{e['generation']}' "
f"'gs://{BUCKET}/{e['path']}'")
print(f"{run_id} {e['result']:<7} {e['path']} gen={e.get('generation')} {e['size_bytes']}B{restore}")
def main():
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("run_id", nargs="?")
parser.add_argument("--find")
args = parser.parse_args()
bucket = _bucket()
if args.find:
find(bucket, args.find)
elif args.run_id:
show_run(bucket, args.run_id)
else:
list_runs(bucket)
if __name__ == "__main__":
sys.exit(main())