Files
karaoke-gen/backend/services/input_persistence.py
T
Andrew Beveridge ☄️andClaude Opus 5.5 983f6ecdcd fix(storage): keep user-uploaded inputs forever; uploads/ expires after 180d not 7d (v0.268.2) (#1137)
## Summary
`uploads/` had a bucket lifecycle **delete after 7 days**, and user
inputs were only partly copied out of it. The only copy step was the
audio worker (stem separation) copying the mix to `jobs/{id}/input/`,
best-effort. Everything else stayed in `uploads/`:
- bring-your-own-instrumental / tenant jobs, which skip separation;
- the uploaded instrumental, which was only staged at render time;
- lyrics and style uploads, which were never copied.

Result:
- randy-vild re-renders 404'd on their own input (2026-10-08). That data
was restored from Andrew's Dropbox.
- Firestore shows 64 completed Jan–Feb 2026 consumer jobs, plus
BYO-instrumental jobs, whose inputs are gone permanently.

## Changes
- **`backend/services/input_persistence.py`** (new):
- Server-side copies `uploads/{id}/**` to `jobs/{id}/input/**` and
repoints every job field that referenced the upload:
`input_media_gcs_path`, `existing_instrumental_gcs_path`,
`lyrics_file_gcs_path`, `style_params_gcs_path`, plus any matching value
under `file_urls` / `state_data` / `style_assets`.
  - Idempotent: unchanged copies are skipped.
- The audio, lyrics, screens, render-video and video workers all call it
on entry, via `asyncio.to_thread`. On failure it logs an ERROR, which
alerts, but doesn't fail the job.
- **`POST /api/admin/persist-uploads`** (dry run by default): backfill
for jobs no worker will touch again. Lookup failures are isolated per
job.
- **GCE encoder:** prefers the explicitly downloaded top-level
`existing_instrumental.*`. The recursive glob could otherwise return the
raw, unconformed copy now kept under `input/audio/`.
- **Pulumi:** `uploads/` lifecycle goes from 7 to **180 days**; `temp/`
stays at 7. **Already applied** and verified directly on the bucket.
- **Docs:** LESSONS-LEARNED, DISASTER-RECOVERY (new user-inputs row; the
stale "30d noncurrent" figure corrected to 7d), API, README.

## Verification
- `make test` all green.
- 16 new tests:
- copy mapping, repointing, idempotency, re-copy when a kept copy
differs, already-expired paths left alone;
  - the worker wrapper never raises;
- an AST guard that every pipeline worker calls the copy step via
`to_thread`;
  - backfill dry run / apply / error isolation;
  - the encoder's instrumental preference.
- Local CodeRabbit: 2 findings, both fixed (lyrics path, per-job
lookup); re-review clean.

## After merge
Run `POST /api/admin/persist-uploads` (dry run, then `?dry_run=false`)
to copy the inputs of the 51 jobs that still have objects in `uploads/`.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 17:33:06 -04:00

107 lines
4.4 KiB
Python

"""
Keep user-uploaded inputs forever: copy ``uploads/{job_id}/**`` into ``jobs/{job_id}/input/``.
Uploads land under ``uploads/{job_id}/`` (signed-URL / multipart / downloader
destinations). That prefix has a bucket lifecycle delete, and user inputs (the
mix, a bring-your-own instrumental and its conformed copy, a lyrics file, style
uploads) are irreplaceable: re-renders, regenerations after the storage-retention
purge and late reviews all need them again. Losing them made tenant re-renders
fail (2026-10-08: every randy-vild job left in review for >7 days had lost its
mix and instrumental).
``persist_job_inputs`` server-side copies every object under ``uploads/{job_id}/``
to ``jobs/{job_id}/input/<same relative path>`` (``input/`` is KEEP-forever in
``storage_retention``) and repoints every job field that referenced the
``uploads/`` copy. It is idempotent and cheap when there's nothing new, so each
worker calls it on entry: inputs uploaded later in a job's life (conformed
instrumentals, style uploads, review-time uploads) are caught by the next worker.
"""
import logging
from typing import Any, Dict, Optional, Tuple
logger = logging.getLogger(__name__)
UPLOADS_PREFIX = "uploads/"
def uploads_prefix(job_id: str) -> str:
return f"{UPLOADS_PREFIX}{job_id}/"
def persisted_path(job_id: str, path: str) -> Optional[str]:
"""``uploads/{job_id}/a/b.wav`` -> ``jobs/{job_id}/input/a/b.wav`` (None if not this job's upload)."""
prefix = uploads_prefix(job_id)
if not isinstance(path, str) or not path.startswith(prefix) or path == prefix:
return None
return f"jobs/{job_id}/input/{path[len(prefix):]}"
def _rewrites(value: Any, mapping: Dict[str, str], path: str, out: Dict[str, str]) -> None:
"""Collect dotted Firestore field paths whose string value is a mapped upload path."""
if isinstance(value, str):
if value in mapping:
out[path] = mapping[value]
elif isinstance(value, dict):
for key, child in value.items():
if isinstance(key, str) and "." not in key and "`" not in key:
_rewrites(child, mapping, f"{path}.{key}" if path else key, out)
# Job fields that can hold an input path (top-level, plus everything under these maps).
_SCALAR_FIELDS = (
"input_media_gcs_path", "existing_instrumental_gcs_path", "lyrics_file_gcs_path", "style_params_gcs_path",
)
_MAP_FIELDS = ("file_urls", "state_data", "style_assets")
def persist_job_inputs(job, storage, job_manager) -> Tuple[int, Dict[str, str]]:
"""Copy this job's uploads into ``jobs/{id}/input/`` and repoint the job.
Returns ``(objects_copied, field_updates)``. Raises on storage/Firestore errors.
"""
job_id = job.job_id
copied = 0
mapping: Dict[str, str] = {}
for blob in storage.bucket.list_blobs(prefix=uploads_prefix(job_id)):
dest = persisted_path(job_id, blob.name)
if not dest:
continue
mapping[blob.name] = dest
existing = storage.bucket.get_blob(dest)
if existing is not None and existing.md5_hash == blob.md5_hash and existing.size == blob.size:
continue
storage.bucket.copy_blob(blob, storage.bucket, dest)
copied += 1
updates: Dict[str, str] = {}
for field in _SCALAR_FIELDS:
value = getattr(job, field, None)
if value in mapping:
updates[field] = mapping[value]
for field in _MAP_FIELDS:
_rewrites(getattr(job, field, None) or {}, mapping, field, updates)
if updates:
job_manager.update_job(job_id, dict(updates))
# Keep the in-memory job consistent for the caller.
for field in _SCALAR_FIELDS:
if field in updates:
setattr(job, field, updates[field])
if copied or updates:
logger.info(f"Job {job_id}: persisted {copied} upload(s) to jobs/{job_id}/input/; repointed {sorted(updates)}")
return copied, updates
def ensure_job_inputs_persisted(job, storage, job_manager, worker: str = "") -> None:
"""Worker-entry wrapper: never fails the job, but logs an ERROR (alerts) on failure.
The ``uploads/`` lifecycle gives months to retry, and every later worker retries.
"""
try:
persist_job_inputs(job, storage, job_manager)
except Exception as e: # noqa: BLE001
logger.error(
f"Job {getattr(job, 'job_id', '?')}: failed to persist uploaded inputs to jobs/ "
f"({worker or 'worker'}): {e}"
)