mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary
`uploads/` had a bucket lifecycle **delete after 7 days**, and user
inputs were only partly copied out of it. The only copy step was the
audio worker (stem separation) copying the mix to `jobs/{id}/input/`,
best-effort. Everything else stayed in `uploads/`:
- bring-your-own-instrumental / tenant jobs, which skip separation;
- the uploaded instrumental, which was only staged at render time;
- lyrics and style uploads, which were never copied.
Result:
- randy-vild re-renders 404'd on their own input (2026-10-08). That data
was restored from Andrew's Dropbox.
- Firestore shows 64 completed Jan–Feb 2026 consumer jobs, plus
BYO-instrumental jobs, whose inputs are gone permanently.
## Changes
- **`backend/services/input_persistence.py`** (new):
- Server-side copies `uploads/{id}/**` to `jobs/{id}/input/**` and
repoints every job field that referenced the upload:
`input_media_gcs_path`, `existing_instrumental_gcs_path`,
`lyrics_file_gcs_path`, `style_params_gcs_path`, plus any matching value
under `file_urls` / `state_data` / `style_assets`.
- Idempotent: unchanged copies are skipped.
- The audio, lyrics, screens, render-video and video workers all call it
on entry, via `asyncio.to_thread`. On failure it logs an ERROR, which
alerts, but doesn't fail the job.
- **`POST /api/admin/persist-uploads`** (dry run by default): backfill
for jobs no worker will touch again. Lookup failures are isolated per
job.
- **GCE encoder:** prefers the explicitly downloaded top-level
`existing_instrumental.*`. The recursive glob could otherwise return the
raw, unconformed copy now kept under `input/audio/`.
- **Pulumi:** `uploads/` lifecycle goes from 7 to **180 days**; `temp/`
stays at 7. **Already applied** and verified directly on the bucket.
- **Docs:** LESSONS-LEARNED, DISASTER-RECOVERY (new user-inputs row; the
stale "30d noncurrent" figure corrected to 7d), API, README.
## Verification
- `make test` all green.
- 16 new tests:
- copy mapping, repointing, idempotency, re-copy when a kept copy
differs, already-expired paths left alone;
- the worker wrapper never raises;
- an AST guard that every pipeline worker calls the copy step via
`to_thread`;
- backfill dry run / apply / error isolation;
- the encoder's instrumental preference.
- Local CodeRabbit: 2 findings, both fixed (lyrics path, per-job
lookup); re-review clean.
## After merge
Run `POST /api/admin/persist-uploads` (dry run, then `?dry_run=false`)
to copy the inputs of the 51 jobs that still have objects in `uploads/`.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
107 lines
4.4 KiB
Python
107 lines
4.4 KiB
Python
"""
|
|
Keep user-uploaded inputs forever: copy ``uploads/{job_id}/**`` into ``jobs/{job_id}/input/``.
|
|
|
|
Uploads land under ``uploads/{job_id}/`` (signed-URL / multipart / downloader
|
|
destinations). That prefix has a bucket lifecycle delete, and user inputs (the
|
|
mix, a bring-your-own instrumental and its conformed copy, a lyrics file, style
|
|
uploads) are irreplaceable: re-renders, regenerations after the storage-retention
|
|
purge and late reviews all need them again. Losing them made tenant re-renders
|
|
fail (2026-10-08: every randy-vild job left in review for >7 days had lost its
|
|
mix and instrumental).
|
|
|
|
``persist_job_inputs`` server-side copies every object under ``uploads/{job_id}/``
|
|
to ``jobs/{job_id}/input/<same relative path>`` (``input/`` is KEEP-forever in
|
|
``storage_retention``) and repoints every job field that referenced the
|
|
``uploads/`` copy. It is idempotent and cheap when there's nothing new, so each
|
|
worker calls it on entry: inputs uploaded later in a job's life (conformed
|
|
instrumentals, style uploads, review-time uploads) are caught by the next worker.
|
|
"""
|
|
import logging
|
|
from typing import Any, Dict, Optional, Tuple
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
UPLOADS_PREFIX = "uploads/"
|
|
|
|
|
|
def uploads_prefix(job_id: str) -> str:
|
|
return f"{UPLOADS_PREFIX}{job_id}/"
|
|
|
|
|
|
def persisted_path(job_id: str, path: str) -> Optional[str]:
|
|
"""``uploads/{job_id}/a/b.wav`` -> ``jobs/{job_id}/input/a/b.wav`` (None if not this job's upload)."""
|
|
prefix = uploads_prefix(job_id)
|
|
if not isinstance(path, str) or not path.startswith(prefix) or path == prefix:
|
|
return None
|
|
return f"jobs/{job_id}/input/{path[len(prefix):]}"
|
|
|
|
|
|
def _rewrites(value: Any, mapping: Dict[str, str], path: str, out: Dict[str, str]) -> None:
|
|
"""Collect dotted Firestore field paths whose string value is a mapped upload path."""
|
|
if isinstance(value, str):
|
|
if value in mapping:
|
|
out[path] = mapping[value]
|
|
elif isinstance(value, dict):
|
|
for key, child in value.items():
|
|
if isinstance(key, str) and "." not in key and "`" not in key:
|
|
_rewrites(child, mapping, f"{path}.{key}" if path else key, out)
|
|
|
|
|
|
# Job fields that can hold an input path (top-level, plus everything under these maps).
|
|
_SCALAR_FIELDS = (
|
|
"input_media_gcs_path", "existing_instrumental_gcs_path", "lyrics_file_gcs_path", "style_params_gcs_path",
|
|
)
|
|
_MAP_FIELDS = ("file_urls", "state_data", "style_assets")
|
|
|
|
|
|
def persist_job_inputs(job, storage, job_manager) -> Tuple[int, Dict[str, str]]:
|
|
"""Copy this job's uploads into ``jobs/{id}/input/`` and repoint the job.
|
|
|
|
Returns ``(objects_copied, field_updates)``. Raises on storage/Firestore errors.
|
|
"""
|
|
job_id = job.job_id
|
|
copied = 0
|
|
mapping: Dict[str, str] = {}
|
|
for blob in storage.bucket.list_blobs(prefix=uploads_prefix(job_id)):
|
|
dest = persisted_path(job_id, blob.name)
|
|
if not dest:
|
|
continue
|
|
mapping[blob.name] = dest
|
|
existing = storage.bucket.get_blob(dest)
|
|
if existing is not None and existing.md5_hash == blob.md5_hash and existing.size == blob.size:
|
|
continue
|
|
storage.bucket.copy_blob(blob, storage.bucket, dest)
|
|
copied += 1
|
|
|
|
updates: Dict[str, str] = {}
|
|
for field in _SCALAR_FIELDS:
|
|
value = getattr(job, field, None)
|
|
if value in mapping:
|
|
updates[field] = mapping[value]
|
|
for field in _MAP_FIELDS:
|
|
_rewrites(getattr(job, field, None) or {}, mapping, field, updates)
|
|
|
|
if updates:
|
|
job_manager.update_job(job_id, dict(updates))
|
|
# Keep the in-memory job consistent for the caller.
|
|
for field in _SCALAR_FIELDS:
|
|
if field in updates:
|
|
setattr(job, field, updates[field])
|
|
if copied or updates:
|
|
logger.info(f"Job {job_id}: persisted {copied} upload(s) to jobs/{job_id}/input/; repointed {sorted(updates)}")
|
|
return copied, updates
|
|
|
|
|
|
def ensure_job_inputs_persisted(job, storage, job_manager, worker: str = "") -> None:
|
|
"""Worker-entry wrapper: never fails the job, but logs an ERROR (alerts) on failure.
|
|
|
|
The ``uploads/`` lifecycle gives months to retry, and every later worker retries.
|
|
"""
|
|
try:
|
|
persist_job_inputs(job, storage, job_manager)
|
|
except Exception as e: # noqa: BLE001
|
|
logger.error(
|
|
f"Job {getattr(job, 'job_id', '?')}: failed to persist uploaded inputs to jobs/ "
|
|
f"({worker or 'worker'}): {e}"
|
|
)
|