mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary Andrew approved this follow-up to #1125 and #1126. It turns on real deletion, adds an audit log of every deleted object, and adds an orphan-folder cleanup mode. ### Real deletion on `STORAGE_RETENTION_DRY_RUN=false` and `STORAGE_RETENTION_MAX_JOBS_PER_RUN=50` are added to the backend deploy's `--set-env-vars`. The API service is the only thing that reads them: it passes `--dry-run` to `storage-retention-job`. That keeps the setting in one place, so a later deploy can't silently revert it. ### Deletion audit log - `storage-retention/deletion-logs/<ts>-<mode>.jsonl` has one line per deleted or failed object: timestamp, run_id, mode, job_id, path, size_bytes, generation, category and result. - The log is rewritten after every job or folder, so a crash still leaves a complete record. Each run also writes `<ts>-<mode>.summary.json`. - Deletes are pinned to the logged generation (`if_generation_match`). - `assert_deletable` refuses any path outside `jobs/{id}/`, so `storage-retention/` is never touched. No lifecycle rule matches live objects there. - The Firestore manifest on each job doc now also records generations and the log path. ### Orphan cleanup - `POST /api/internal/storage-retention/orphans` runs on the same Cloud Run job with `--mode orphans`. - It targets `jobs/{id}/` folders with no doc in `jobs`, `jobs-dev` or `youtube_upload_queue`, re-checked live per folder against a fresh listing. - It skips any folder with an object modified in the last 30 days, keeps `input/`, and deletes the rest. - `encoding_worker_jobs` doesn't count as a record: it's the encoder's expiring result cache, which deleted jobs leave behind. - It is a dry-run unless called with `dry_run=false&confirm=delete-orphan-job-folders`, and the scheduler never runs it. ### Docs - `docs/STORAGE-RETENTION.md`: run modes, log locations, jq review commands and restore one-liners (noncurrent `cp #gen` and soft-delete `restore`). - `scripts/storage_retention_report.py`: list runs, summarise a run, or find which run deleted a path or job. ## Testing New tests cover the log contents, generation-pinned deletes, per-job flushing across a crash, the path guard, orphan planning (input kept, 30-day skip, live record re-check, batch limit), and the endpoint and worker modes. Full backend suite: 5774 passed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added admin-triggered orphan-folder cleanup, with dry-run enabled by default, optional folder limits, and explicit confirmation required for deletion. * Added deletion reports and audit logs covering removed objects, reclaimed space, and failures, plus a read-only tool to review runs and search records. * **Bug Fixes** * Added safeguards to prevent deleting changed objects or files outside their intended job folders. * **Documentation** * Expanded storage-retention guidance with recovery details, review commands, and cleanup instructions. * **Chores** * Set the production cleanup limit to 50 jobs per run and updated the package version. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>