Files
karaoke-gen/docs/STORAGE-RETENTION.md
T
Andrew Beveridge ☄️andClaude Opus 5.5 3ecea6f2e7 feat(storage): enable real purge with deletion audit log and orphan cleanup mode (v0.266.0) (#1127)
## Summary
Andrew approved this follow-up to #1125 and #1126. It turns on real
deletion, adds an audit log of every deleted object, and adds an
orphan-folder cleanup mode.

### Real deletion on
`STORAGE_RETENTION_DRY_RUN=false` and
`STORAGE_RETENTION_MAX_JOBS_PER_RUN=50` are added to the backend
deploy's `--set-env-vars`. The API service is the only thing that reads
them: it passes `--dry-run` to `storage-retention-job`. That keeps the
setting in one place, so a later deploy can't silently revert it.

### Deletion audit log
- `storage-retention/deletion-logs/<ts>-<mode>.jsonl` has one line per
deleted or failed object: timestamp, run_id, mode, job_id, path,
size_bytes, generation, category and result.
- The log is rewritten after every job or folder, so a crash still
leaves a complete record. Each run also writes
`<ts>-<mode>.summary.json`.
- Deletes are pinned to the logged generation (`if_generation_match`).
- `assert_deletable` refuses any path outside `jobs/{id}/`, so
`storage-retention/` is never touched. No lifecycle rule matches live
objects there.
- The Firestore manifest on each job doc now also records generations
and the log path.

### Orphan cleanup
- `POST /api/internal/storage-retention/orphans` runs on the same Cloud
Run job with `--mode orphans`.
- It targets `jobs/{id}/` folders with no doc in `jobs`, `jobs-dev` or
`youtube_upload_queue`, re-checked live per folder against a fresh
listing.
- It skips any folder with an object modified in the last 30 days, keeps
`input/`, and deletes the rest.
- `encoding_worker_jobs` doesn't count as a record: it's the encoder's
expiring result cache, which deleted jobs leave behind.
- It is a dry-run unless called with
`dry_run=false&confirm=delete-orphan-job-folders`, and the scheduler
never runs it.

### Docs
- `docs/STORAGE-RETENTION.md`: run modes, log locations, jq review
commands and restore one-liners (noncurrent `cp #gen` and soft-delete
`restore`).
- `scripts/storage_retention_report.py`: list runs, summarise a run, or
find which run deleted a path or job.

## Testing
New tests cover the log contents, generation-pinned deletes, per-job
flushing across a crash, the path guard, orphan planning (input kept,
30-day skip, live record re-check, batch limit), and the endpoint and
worker modes. Full backend suite: 5774 passed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added admin-triggered orphan-folder cleanup, with dry-run enabled by
default, optional folder limits, and explicit confirmation required for
deletion.
* Added deletion reports and audit logs covering removed objects,
reclaimed space, and failures, plus a read-only tool to review runs and
search records.
* **Bug Fixes**
* Added safeguards to prevent deleting changed objects or files outside
their intended job folders.
* **Documentation**
* Expanded storage-retention guidance with recovery details, review
commands, and cleanup instructions.
* **Chores**
* Set the production cleanup limit to 50 jobs per run and updated the
package version.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 11:54:05 -04:00

4.3 KiB

Storage retention: operations

Policy + design: docs/ARCHITECTURE.md § Storage Retention. Code: backend/services/storage_retention.py, backend/workers/storage_retention_worker.py.

Runs

Mode Trigger Deletes
job_purge Cloud Scheduler storage-retention-daily (10:30 UTC) → POST /api/internal/storage-retention/run → Cloud Run Job storage-retention-job (us-east4) Regenerable files of jobs completed >30 days ago, up to STORAGE_RETENTION_MAX_JOBS_PER_RUN (50) per run. Real deletion is on (STORAGE_RETENTION_DRY_RUN=false in the CI --set-env-vars of the backend deploy — the API passes the flag to the job; that's the only place it's set).
orphan Manual: POST /api/internal/storage-retention/orphans (dry-run) / ?dry_run=false&confirm=delete-orphan-job-folders Everything except input/ in jobs/{id}/ folders with no record in jobs, jobs-dev or youtube_upload_queue (checked live per folder) and nothing modified in the last 30 days.

Manual trigger (admin token):

TOK=$(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke | cut -d, -f1)
curl -X POST https://api.nomadkaraoke.com/api/internal/storage-retention/run -H "X-Admin-Token: $TOK"
curl -X POST https://api.nomadkaraoke.com/api/internal/storage-retention/orphans -H "X-Admin-Token: $TOK"   # dry-run

Where the record is

  • gs://karaoke-gen-storage-nomadkaraoke/storage-retention/reports/<ts>-{dry-run,run}.json, <ts>-orphans-{dry-run,run}.json: the plan/outcome of each pass (dry-runs included).
  • gs://karaoke-gen-storage-nomadkaraoke/storage-retention/deletion-logs/<ts>-<mode>.jsonl: one line per object a real run deleted (or failed to): timestamp, run_id, mode (job_purge|orphan), job_id, path, size_bytes, generation, category (finals|videos|stems|review-audio|previews|screens-mov|encoded|quick|orphan-nonInput), result (deleted|failed), error. Rewritten after every job/folder, so a crash leaves a complete record up to the last finished job.
  • .../deletion-logs/<ts>-<mode>.summary.json: totals by category, job count, GiB, failures.
  • Each purged job's Firestore doc: storage_purge (files + generations + deletion_log), renders_purged_at, stems_purged_at.

The storage-retention/ prefix is never touched by retention code (it only deletes inside jobs/{id}/, enforced by assert_deletable) and no lifecycle rule matches live objects there (the rules are: age 7 on temp/+uploads/, and noncurrent versions after 7 days). Logs are kept forever.

Reviewing

B=gs://karaoke-gen-storage-nomadkaraoke/storage-retention/deletion-logs
gcloud storage ls "$B/"                                        # all runs
RUN=20261004T150000Z-job_purge                                 # pick one from the listing
gcloud storage cat "$B/$RUN.summary.json" | jq .               # one run's totals
gcloud storage cat "$B/$RUN.jsonl" | jq -s 'map(select(.result=="deleted")) | group_by(.category)
  | map({category: .[0].category, objects: length, gib: (map(.size_bytes)|add/1073741824)})'
gcloud storage cat "$B/$RUN.jsonl" | jq -s 'group_by(.job_id) | map({job: .[0].job_id,
  gib: (map(.size_bytes)|add/1073741824)}) | sort_by(-.gib)'   # per job
gcloud storage cat "$B/$RUN.jsonl" | jq -c 'select(.result=="failed")'
JOB=abc12345
gcloud storage cat "$B/*.jsonl" | jq -c --arg j "$JOB" 'select(.job_id==$j)'   # everything deleted for a job

Or: python scripts/storage_retention_report.py (all runs), ... <run_id> (one run, per category and job), ... --find <job_id|path> (which run deleted it, with a restore command).

Restoring

The bucket keeps deleted objects as noncurrent versions for 7 days, then soft delete keeps them 7 more days. With the generation from the log:

OBJ='jobs/abc12345/finals/lossy_4k_mp4.mp4'; GEN=1790000000000000   # from a result=="deleted" log line
# within ~7 days (noncurrent version):
gcloud storage cp --no-clobber "gs://karaoke-gen-storage-nomadkaraoke/$OBJ#$GEN" "gs://karaoke-gen-storage-nomadkaraoke/$OBJ"
# days 7-14 (soft-deleted):
gcloud storage restore "gs://karaoke-gen-storage-nomadkaraoke/$OBJ#$GEN"

After restoring a purged job's files, prefer simply using "Regenerate video" (or POST /api/admin/jobs/{id}/regenerate) — it rebuilds everything and clears the purge markers.