fix(storage): keep user-uploaded inputs forever; uploads/ expires after 180d not 7d (v0.268.2) (#1137)

## Summary
`uploads/` had a bucket lifecycle **delete after 7 days**, and user
inputs were only partly copied out of it. The only copy step was the
audio worker (stem separation) copying the mix to `jobs/{id}/input/`,
best-effort. Everything else stayed in `uploads/`:
- bring-your-own-instrumental / tenant jobs, which skip separation;
- the uploaded instrumental, which was only staged at render time;
- lyrics and style uploads, which were never copied.

Result:
- randy-vild re-renders 404'd on their own input (2026-10-08). That data
was restored from Andrew's Dropbox.
- Firestore shows 64 completed Jan–Feb 2026 consumer jobs, plus
BYO-instrumental jobs, whose inputs are gone permanently.

## Changes
- **`backend/services/input_persistence.py`** (new):
- Server-side copies `uploads/{id}/**` to `jobs/{id}/input/**` and
repoints every job field that referenced the upload:
`input_media_gcs_path`, `existing_instrumental_gcs_path`,
`lyrics_file_gcs_path`, `style_params_gcs_path`, plus any matching value
under `file_urls` / `state_data` / `style_assets`.
  - Idempotent: unchanged copies are skipped.
- The audio, lyrics, screens, render-video and video workers all call it
on entry, via `asyncio.to_thread`. On failure it logs an ERROR, which
alerts, but doesn't fail the job.
- **`POST /api/admin/persist-uploads`** (dry run by default): backfill
for jobs no worker will touch again. Lookup failures are isolated per
job.
- **GCE encoder:** prefers the explicitly downloaded top-level
`existing_instrumental.*`. The recursive glob could otherwise return the
raw, unconformed copy now kept under `input/audio/`.
- **Pulumi:** `uploads/` lifecycle goes from 7 to **180 days**; `temp/`
stays at 7. **Already applied** and verified directly on the bucket.
- **Docs:** LESSONS-LEARNED, DISASTER-RECOVERY (new user-inputs row; the
stale "30d noncurrent" figure corrected to 7d), API, README.

## Verification
- `make test` all green.
- 16 new tests:
- copy mapping, repointing, idempotency, re-copy when a kept copy
differs, already-expired paths left alone;
  - the worker wrapper never raises;
- an AST guard that every pipeline worker calls the copy step via
`to_thread`;
  - backfill dry run / apply / error isolation;
  - the encoder's instrumental preference.
- Local CodeRabbit: 2 findings, both fixed (lyrics path, per-job
lookup); re-review clean.

## After merge
Run `POST /api/admin/persist-uploads` (dry run, then `?dry_run=false`)
to copy the inputs of the 51 jobs that still have objects in `uploads/`.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Opus 5.5 authored and GitHub committed 2026-10-08 17:33:06 -04:00
1 parent 60be7606a3
commit 983f6ecdcd
16 files changed
+479 -9

No files matched your search

+42
View File
@@ -4505,3 +4505,45 @@ async def action_community_review(
status="made", request_id=request_id,
message=f"Making our version (job {result.get('job_id')}).",
)
class PersistUploadsResponse(BaseModel):
dry_run: bool
jobs: Dict[str, Any] # job_id -> {"copied", "updated"} | {"would_copy"} | {"error"} | {"orphan"}
@router.post("/persist-uploads", response_model=PersistUploadsResponse)
def persist_uploads(dry_run: bool = True, auth_data: AuthResult = Depends(require_admin)):
"""Backfill: copy every job's ``uploads/{job_id}/**`` into ``jobs/{job_id}/input/``.
Same operation the workers run on entry (``input_persistence``), for jobs that
no worker will touch again. Folders with no job document (abandoned uploads)
are reported as ``orphan`` and left in place. Plain ``def``: blocking GCS and
Firestore calls run in the threadpool.
"""
from backend.services.input_persistence import persist_job_inputs, persisted_path
storage = StorageService()
job_manager = JobManager()
by_job: Dict[str, List[str]] = {}
for blob in storage.bucket.list_blobs(prefix="uploads/"):
parts = blob.name.split("/")
if len(parts) > 2 and parts[1]:
by_job.setdefault(parts[1], []).append(blob.name)
results: Dict[str, Any] = {}
for job_id, names in sorted(by_job.items()):
try:
job = job_manager.get_job(job_id)
if job is None:
results[job_id] = {"orphan": len(names)}
continue
if dry_run:
results[job_id] = {"would_copy": {n: persisted_path(job_id, n) for n in names}}
continue
copied, updates = persist_job_inputs(job, storage, job_manager)
results[job_id] = {"copied": copied, "updated": updates}
except Exception as e: # noqa: BLE001
logger.error(f"persist-uploads: job {job_id} failed: {e}")
results[job_id] = {"error": str(e)}
return PersistUploadsResponse(dry_run=dry_run, jobs=results)
+1 -1
View File
@@ -1664,7 +1664,7 @@ async def mark_uploads_complete(
logger.warning(f"Duration validation failed with error: {e}. Proceeding without validation.")
# Don't block the job if we can't validate - the video worker will fail more gracefully
# uploads/ expires after 7 days: keep the user's instrumental at the job
# uploads/ expires (bucket lifecycle): keep the user's instrumental at the job
# root (where the video pipeline stages it anyway) so later re-renders,
# regenerates and edits still have it.
if (update_data.get('existing_instrumental_gcs_path') or '').startswith('uploads/'):
+7 -2
View File
@@ -867,8 +867,13 @@ def resolve_instrumental(work_dir: Path, config: dict):
if existing_instrumental:
# User-provided instrumental uploaded at job creation
# Downloaded to work_dir by process_job before run_encoding is called
instrumental = find_file(
# Downloaded to work_dir by process_job before run_encoding is called.
# Prefer that top-level copy (it's exactly config["existing_instrumental"]):
# the job folder also holds persisted inputs (input/audio/existing_instrumental.*
# = the raw upload, input/conformed/... = its aligned copy), which the
# recursive glob below could return instead.
top_level = sorted(work_dir.glob("existing_instrumental.*"))
instrumental = top_level[0] if top_level else find_file(
work_dir,
"*existing_instrumental*", "*Instrumental User*",
)
+106
View File
@@ -0,0 +1,106 @@
"""
Keep user-uploaded inputs forever: copy ``uploads/{job_id}/**`` into ``jobs/{job_id}/input/``.
Uploads land under ``uploads/{job_id}/`` (signed-URL / multipart / downloader
destinations). That prefix has a bucket lifecycle delete, and user inputs (the
mix, a bring-your-own instrumental and its conformed copy, a lyrics file, style
uploads) are irreplaceable: re-renders, regenerations after the storage-retention
purge and late reviews all need them again. Losing them made tenant re-renders
fail (2026-10-08: every randy-vild job left in review for >7 days had lost its
mix and instrumental).
``persist_job_inputs`` server-side copies every object under ``uploads/{job_id}/``
to ``jobs/{job_id}/input/<same relative path>`` (``input/`` is KEEP-forever in
``storage_retention``) and repoints every job field that referenced the
``uploads/`` copy. It is idempotent and cheap when there's nothing new, so each
worker calls it on entry: inputs uploaded later in a job's life (conformed
instrumentals, style uploads, review-time uploads) are caught by the next worker.
"""
import logging
from typing import Any, Dict, Optional, Tuple
logger = logging.getLogger(__name__)
UPLOADS_PREFIX = "uploads/"
def uploads_prefix(job_id: str) -> str:
return f"{UPLOADS_PREFIX}{job_id}/"
def persisted_path(job_id: str, path: str) -> Optional[str]:
"""``uploads/{job_id}/a/b.wav`` -> ``jobs/{job_id}/input/a/b.wav`` (None if not this job's upload)."""
prefix = uploads_prefix(job_id)
if not isinstance(path, str) or not path.startswith(prefix) or path == prefix:
return None
return f"jobs/{job_id}/input/{path[len(prefix):]}"
def _rewrites(value: Any, mapping: Dict[str, str], path: str, out: Dict[str, str]) -> None:
"""Collect dotted Firestore field paths whose string value is a mapped upload path."""
if isinstance(value, str):
if value in mapping:
out[path] = mapping[value]
elif isinstance(value, dict):
for key, child in value.items():
if isinstance(key, str) and "." not in key and "`" not in key:
_rewrites(child, mapping, f"{path}.{key}" if path else key, out)
# Job fields that can hold an input path (top-level, plus everything under these maps).
_SCALAR_FIELDS = (
"input_media_gcs_path", "existing_instrumental_gcs_path", "lyrics_file_gcs_path", "style_params_gcs_path",
)
_MAP_FIELDS = ("file_urls", "state_data", "style_assets")
def persist_job_inputs(job, storage, job_manager) -> Tuple[int, Dict[str, str]]:
"""Copy this job's uploads into ``jobs/{id}/input/`` and repoint the job.
Returns ``(objects_copied, field_updates)``. Raises on storage/Firestore errors.
"""
job_id = job.job_id
copied = 0
mapping: Dict[str, str] = {}
for blob in storage.bucket.list_blobs(prefix=uploads_prefix(job_id)):
dest = persisted_path(job_id, blob.name)
if not dest:
continue
mapping[blob.name] = dest
existing = storage.bucket.get_blob(dest)
if existing is not None and existing.md5_hash == blob.md5_hash and existing.size == blob.size:
continue
storage.bucket.copy_blob(blob, storage.bucket, dest)
copied += 1
updates: Dict[str, str] = {}
for field in _SCALAR_FIELDS:
value = getattr(job, field, None)
if value in mapping:
updates[field] = mapping[value]
for field in _MAP_FIELDS:
_rewrites(getattr(job, field, None) or {}, mapping, field, updates)
if updates:
job_manager.update_job(job_id, dict(updates))
# Keep the in-memory job consistent for the caller.
for field in _SCALAR_FIELDS:
if field in updates:
setattr(job, field, updates[field])
if copied or updates:
logger.info(f"Job {job_id}: persisted {copied} upload(s) to jobs/{job_id}/input/; repointed {sorted(updates)}")
return copied, updates
def ensure_job_inputs_persisted(job, storage, job_manager, worker: str = "") -> None:
"""Worker-entry wrapper: never fails the job, but logs an ERROR (alerts) on failure.
The ``uploads/`` lifecycle gives months to retry, and every later worker retries.
"""
try:
persist_job_inputs(job, storage, job_manager)
except Exception as e: # noqa: BLE001
logger.error(
f"Job {getattr(job, 'job_id', '?')}: failed to persist uploaded inputs to jobs/ "
f"({worker or 'worker'}): {e}"
)
+255
View File
@@ -0,0 +1,255 @@
"""
User-uploaded inputs must outlive the uploads/ lifecycle delete.
2026-10-08: randy-vild jobs that skipped the audio worker's copy (bring-your-own
instrumental) or sat in review for >7 days lost their mix and instrumental, so
re-renders failed with a 404 on uploads/... . Every pipeline worker now copies
uploads/{job_id}/** to jobs/{job_id}/input/ on entry and repoints the job.
"""
import ast
import hashlib
from pathlib import Path
from types import SimpleNamespace
from unittest.mock import MagicMock, patch
import pytest
from fastapi import FastAPI
from fastapi.testclient import TestClient
from backend.services.input_persistence import (
ensure_job_inputs_persisted,
persist_job_inputs,
persisted_path,
)
class FakeBlob:
def __init__(self, name, data):
self.name = name
self.data = data
self.size = len(data)
self.md5_hash = hashlib.md5(data).hexdigest()
class FakeBucket:
def __init__(self, objects):
self.objects = {n: FakeBlob(n, d) for n, d in objects.items()}
self.copies = []
def list_blobs(self, prefix=""):
return [b for n, b in sorted(self.objects.items()) if n.startswith(prefix)]
def get_blob(self, name):
return self.objects.get(name)
def copy_blob(self, blob, bucket, dest):
self.copies.append((blob.name, dest))
self.objects[dest] = FakeBlob(dest, blob.data)
def _job(**fields):
base = dict(
job_id="j1",
input_media_gcs_path="uploads/j1/audio/01-Song Mix.wav",
existing_instrumental_gcs_path="uploads/j1/conformed/existing_instrumental.flac",
style_params_gcs_path=None,
lyrics_file_gcs_path="uploads/j1/lyrics/user_lyrics.txt",
file_urls={"input": {"audio": "uploads/j1/audio/01-Song Mix.wav"}, "stems": {"x": "jobs/j1/stems/x.flac"}},
state_data={"instrumental_conformed": {"original_gcs_path": "uploads/j1/audio/existing_instrumental.wav", "correlation": 0.8}},
style_assets={},
)
base.update(fields)
return SimpleNamespace(**base)
UPLOADS = {
"uploads/j1/audio/01-Song Mix.wav": b"mix",
"uploads/j1/audio/existing_instrumental.wav": b"inst",
"uploads/j1/conformed/existing_instrumental.flac": b"conformed",
"uploads/j1/lyrics/user_lyrics.txt": b"la la",
"uploads/j10/audio/other.wav": b"another job", # prefix-sibling must not be touched
}
def test_persisted_path_keeps_relative_layout_and_only_maps_this_job():
assert persisted_path("j1", "uploads/j1/audio/a b.wav") == "jobs/j1/input/audio/a b.wav"
assert persisted_path("j1", "uploads/j10/audio/x.wav") is None
assert persisted_path("j1", "jobs/j1/input/x.wav") is None
assert persisted_path("j1", "uploads/j1/") is None
def test_copies_every_upload_and_repoints_every_reference():
bucket = FakeBucket(UPLOADS)
jm = MagicMock()
job = _job()
copied, updates = persist_job_inputs(job, SimpleNamespace(bucket=bucket), jm)
assert copied == 4
assert sorted(dest for _src, dest in bucket.copies) == [
"jobs/j1/input/audio/01-Song Mix.wav",
"jobs/j1/input/audio/existing_instrumental.wav",
"jobs/j1/input/conformed/existing_instrumental.flac",
"jobs/j1/input/lyrics/user_lyrics.txt",
]
assert updates == {
"input_media_gcs_path": "jobs/j1/input/audio/01-Song Mix.wav",
"existing_instrumental_gcs_path": "jobs/j1/input/conformed/existing_instrumental.flac",
"lyrics_file_gcs_path": "jobs/j1/input/lyrics/user_lyrics.txt",
"file_urls.input.audio": "jobs/j1/input/audio/01-Song Mix.wav",
"state_data.instrumental_conformed.original_gcs_path": "jobs/j1/input/audio/existing_instrumental.wav",
}
jm.update_job.assert_called_once_with("j1", updates)
# in-memory job follows, so the calling worker downloads from the kept copy
assert job.input_media_gcs_path == "jobs/j1/input/audio/01-Song Mix.wav"
assert job.existing_instrumental_gcs_path.startswith("jobs/j1/input/")
def test_idempotent_second_run_copies_nothing():
bucket = FakeBucket(UPLOADS)
jm = MagicMock()
persist_job_inputs(_job(), SimpleNamespace(bucket=bucket), jm)
bucket.copies.clear()
jm.reset_mock()
# Job already repointed (as Firestore would now hold)
job = _job(
input_media_gcs_path="jobs/j1/input/audio/01-Song Mix.wav",
existing_instrumental_gcs_path="jobs/j1/input/conformed/existing_instrumental.flac",
lyrics_file_gcs_path="jobs/j1/input/lyrics/user_lyrics.txt",
file_urls={"input": {"audio": "jobs/j1/input/audio/01-Song Mix.wav"}},
state_data={},
)
copied, updates = persist_job_inputs(job, SimpleNamespace(bucket=bucket), jm)
assert (copied, updates) == (0, {})
assert bucket.copies == []
jm.update_job.assert_not_called()
def test_recopies_when_the_kept_copy_differs():
"""A re-upload under the same name (e.g. a new conformed instrumental) replaces the kept copy."""
bucket = FakeBucket({**UPLOADS, "jobs/j1/input/conformed/existing_instrumental.flac": b"stale"})
persist_job_inputs(_job(), SimpleNamespace(bucket=bucket), MagicMock())
assert bucket.objects["jobs/j1/input/conformed/existing_instrumental.flac"].data == b"conformed"
def test_paths_to_already_expired_uploads_are_left_alone():
bucket = FakeBucket({}) # nothing left in uploads/
jm = MagicMock()
copied, updates = persist_job_inputs(_job(), SimpleNamespace(bucket=bucket), jm)
assert (copied, updates) == (0, {})
jm.update_job.assert_not_called()
def test_style_assets_and_style_params_are_repointed():
bucket = FakeBucket({"uploads/j1/style/style_params.json": b"{}", "uploads/j1/style/bg.png": b"png"})
job = _job(
input_media_gcs_path=None, existing_instrumental_gcs_path=None, lyrics_file_gcs_path=None,
file_urls={}, state_data={}, style_params_gcs_path="uploads/j1/style/style_params.json",
style_assets={"karaoke_background": "uploads/j1/style/bg.png", "font": "themes/nomad/assets/A.ttf"},
)
_copied, updates = persist_job_inputs(job, SimpleNamespace(bucket=bucket), MagicMock())
assert updates == {
"style_params_gcs_path": "jobs/j1/input/style/style_params.json",
"style_assets.karaoke_background": "jobs/j1/input/style/bg.png",
}
def test_worker_wrapper_never_raises_but_logs_an_error(caplog):
storage = SimpleNamespace(bucket=MagicMock(list_blobs=MagicMock(side_effect=RuntimeError("gcs down"))))
with caplog.at_level("ERROR"):
ensure_job_inputs_persisted(_job(), storage, MagicMock(), "lyrics")
assert any("failed to persist uploaded inputs" in r.message and "gcs down" in r.message for r in caplog.records)
@pytest.mark.parametrize(
"module",
["audio_worker", "lyrics_worker", "screens_worker", "render_video_worker", "video_worker"],
)
def test_every_pipeline_worker_persists_inputs_off_the_event_loop(module):
"""Each worker entry calls the persistence step via asyncio.to_thread (blocking GCS/Firestore I/O)."""
source = Path(__file__).resolve().parents[1].joinpath("workers", f"{module}.py").read_text()
calls = [
node for node in ast.walk(ast.parse(source))
if isinstance(node, ast.Call)
and getattr(node.func, "attr", None) == "to_thread"
and node.args and getattr(node.args[0], "id", None) == "ensure_job_inputs_persisted"
]
assert calls, f"{module} must call ensure_job_inputs_persisted on entry"
# --- admin backfill -------------------------------------------------------------------
def _admin_client():
from backend.api.dependencies import AuthResult, UserType, require_admin
from backend.api.routes.admin import router
app = FastAPI()
app.include_router(router, prefix="/api")
app.dependency_overrides[require_admin] = lambda: AuthResult(
is_valid=True, user_type=UserType.ADMIN, remaining_uses=999, message="ok",
user_email="admin@example.com", is_admin=True,
)
return TestClient(app)
def test_backfill_dry_run_reports_without_copying():
bucket = FakeBucket({**UPLOADS, "uploads/gone/audio/x.wav": b"orphan"})
jm = MagicMock()
jm.get_job.side_effect = lambda jid: _job() if jid == "j1" else (_job(job_id="j10") if jid == "j10" else None)
with patch("backend.api.routes.admin.StorageService", return_value=SimpleNamespace(bucket=bucket)), \
patch("backend.api.routes.admin.JobManager", return_value=jm):
resp = _admin_client().post("/api/admin/persist-uploads")
assert resp.status_code == 200
body = resp.json()
assert body["dry_run"] is True
assert body["jobs"]["gone"] == {"orphan": 1}
assert body["jobs"]["j1"]["would_copy"]["uploads/j1/lyrics/user_lyrics.txt"] == "jobs/j1/input/lyrics/user_lyrics.txt"
assert bucket.copies == []
jm.update_job.assert_not_called()
def test_backfill_apply_persists_each_job():
bucket = FakeBucket(UPLOADS)
jm = MagicMock()
jm.get_job.side_effect = lambda jid: _job(job_id=jid, input_media_gcs_path=f"uploads/{jid}/audio/other.wav",
existing_instrumental_gcs_path=None, lyrics_file_gcs_path=None,
file_urls={}, state_data={}) \
if jid == "j10" else _job()
with patch("backend.api.routes.admin.StorageService", return_value=SimpleNamespace(bucket=bucket)), \
patch("backend.api.routes.admin.JobManager", return_value=jm):
body = _admin_client().post("/api/admin/persist-uploads?dry_run=false").json()
assert body["jobs"]["j1"]["copied"] == 4
assert body["jobs"]["j10"] == {"copied": 1, "updated": {"input_media_gcs_path": "jobs/j10/input/audio/other.wav"}}
# --- GCE encoder picks the configured instrumental, not a persisted raw copy -----------
def test_encoder_prefers_the_downloaded_existing_instrumental(tmp_path):
from backend.services.gce_encoding.main import resolve_instrumental
(tmp_path / "input" / "audio").mkdir(parents=True)
(tmp_path / "input" / "conformed").mkdir(parents=True)
(tmp_path / "input" / "audio" / "existing_instrumental.wav").write_bytes(b"raw, unaligned")
(tmp_path / "input" / "conformed" / "existing_instrumental.flac").write_bytes(b"aligned")
(tmp_path / "existing_instrumental.flac").write_bytes(b"configured")
picked = resolve_instrumental(tmp_path, {"existing_instrumental": "jobs/j1/input/conformed/existing_instrumental.flac"})
assert picked == tmp_path / "existing_instrumental.flac"
def test_backfill_keeps_going_when_one_job_lookup_fails():
bucket = FakeBucket(UPLOADS)
jm = MagicMock()
def get_job(jid):
if jid == "j1":
raise RuntimeError("firestore hiccup")
return _job(job_id=jid, input_media_gcs_path=f"uploads/{jid}/audio/other.wav",
existing_instrumental_gcs_path=None, lyrics_file_gcs_path=None, file_urls={}, state_data={})
jm.get_job.side_effect = get_job
with patch("backend.api.routes.admin.StorageService", return_value=SimpleNamespace(bucket=bucket)), \
patch("backend.api.routes.admin.JobManager", return_value=jm):
body = _admin_client().post("/api/admin/persist-uploads?dry_run=false").json()
assert body["jobs"]["j1"] == {"error": "firestore hiccup"}
assert body["jobs"]["j10"]["copied"] == 1
+5 -1
View File
@@ -24,6 +24,7 @@ import time
from typing import Optional, Dict, Any
from backend.services.job_manager import JobManager
from backend.services.input_persistence import ensure_job_inputs_persisted
from backend.services.storage_service import StorageService
from backend.services.audio_search_service import DownloadError
from backend.services.job_health_service import validate_worker_can_run
@@ -183,6 +184,9 @@ async def process_audio_separation(job_id: str) -> bool:
job_log.error(f"Job {job_id} not found in Firestore!")
return False
# Copy uploads/{job_id}/** into jobs/{job_id}/input/ (uploads/ expires) and repoint the job.
await asyncio.to_thread(ensure_job_inputs_persisted, job, storage, job_manager, "audio")
# Validate job status is appropriate for audio worker
# This helps catch bugs where the worker is triggered incorrectly
status_error = validate_worker_can_run("audio_worker", job)
@@ -519,7 +523,7 @@ async def download_audio(
storage.download_file(job.input_media_gcs_path, local_path)
logger.info(f"Job {job_id}: Downloaded uploaded file to {local_path}")
# Persist uploaded audio to jobs/ prefix (uploads/ has 7-day lifecycle)
# Persist uploaded audio to jobs/ prefix (uploads/ is a lifecycle-expiring staging area; see input_persistence)
if job.input_media_gcs_path.startswith("uploads/"):
try:
persistent_path = f"jobs/{job_id}/input/{safe_filename}"
+4
View File
@@ -30,6 +30,7 @@ from pathlib import Path
from backend.models.job import JobStatus
from backend.services.job_manager import JobManager
from backend.services.input_persistence import ensure_job_inputs_persisted
from backend.services.storage_service import StorageService
from backend.services.audio_search_service import DownloadError
from backend.services.job_health_service import validate_worker_can_run
@@ -257,6 +258,9 @@ async def process_lyrics_transcription(job_id: str) -> bool:
job_log.error(f"Job {job_id} not found in Firestore!")
return False
# Copy uploads/{job_id}/** into jobs/{job_id}/input/ (uploads/ expires) and repoint the job.
await asyncio.to_thread(ensure_job_inputs_persisted, job, storage, job_manager, "lyrics")
# Validate job status is appropriate for lyrics worker
# This helps catch bugs where the worker is triggered incorrectly
status_error = validate_worker_can_run("lyrics_worker", job)
+4
View File
@@ -39,6 +39,7 @@ from backend.models.job import JobStatus
from backend.exceptions import InvalidStateTransitionError
from backend.utils.audio_filenames import local_audio_filename
from backend.services.job_manager import JobManager
from backend.services.input_persistence import ensure_job_inputs_persisted
from backend.services.storage_service import StorageService
from backend.services.lyrics_translation import prepare_job_translations
from karaoke_gen.lyrics_transcriber.output.translations import apply_translations
@@ -130,6 +131,9 @@ async def process_render_video(job_id: str) -> bool:
await worker_registry.unregister(job_id, "render-video")
return False
# Copy uploads/{job_id}/** into jobs/{job_id}/input/ (uploads/ expires) and repoint the job.
await asyncio.to_thread(ensure_job_inputs_persisted, job, storage, job_manager, "render_video")
# Hard idempotency gate (worker-side). The legacy HTTP endpoint enforces
# this via _check_worker_idempotency at dispatch time, but the Cloud Run
# Job path invokes this worker directly — and Cloud Run may re-run the
+4
View File
@@ -33,6 +33,7 @@ from pathlib import Path
from backend.models.job import JobStatus
from backend.services.job_manager import JobManager
from backend.services.input_persistence import ensure_job_inputs_persisted
from backend.services.storage_service import StorageService
from backend.services.job_health_service import validate_worker_can_run
from backend.config import get_settings
@@ -100,6 +101,9 @@ async def generate_screens(job_id: str) -> bool:
logger.error(f"[job:{job_id}] Job not found")
return False
# Copy uploads/{job_id}/** into jobs/{job_id}/input/ (uploads/ expires) and repoint the job.
await asyncio.to_thread(ensure_job_inputs_persisted, job, storage, job_manager, "screens")
# Idempotency guard: this worker may be dispatched twice (lyrics primary +
# audio fallback triggers, or a Cloud Tasks redelivery). Once the job has
# advanced past the pre-screens stage, a second dispatch is a duplicate — no-op
+6 -1
View File
@@ -22,6 +22,7 @@ Observability:
- Logs include [job:ID] prefix for easy filtering in Cloud Logging
- Worker start/end timing logged with WORKER_START/WORKER_END markers
"""
import asyncio
import logging
import os
import shutil
@@ -36,6 +37,7 @@ from google.cloud.firestore_v1 import DELETE_FIELD
from backend.models.job import JobStatus
from backend.exceptions import InvalidStateTransitionError
from backend.services.job_manager import JobManager
from backend.services.input_persistence import ensure_job_inputs_persisted
from backend.services.storage_service import StorageService
from backend.services.admin_rerender_service import (
active_admin_rerender,
@@ -261,6 +263,9 @@ async def generate_video_orchestrated(job_id: str) -> bool:
logger.error(f"[job:{job_id}] Job not found")
return False
# Copy uploads/{job_id}/** into jobs/{job_id}/input/ (uploads/ expires) and repoint the job.
await asyncio.to_thread(ensure_job_inputs_persisted, job, storage, job_manager, "video")
# Validate prerequisites
if not _validate_prerequisites(job):
logger.error(f"[job:{job_id}] Prerequisites not met for video generation")
@@ -274,7 +279,7 @@ async def generate_video_orchestrated(job_id: str) -> bool:
job_log.warning(status_error)
return False
# A user-supplied instrumental recorded under uploads/ expires after 7 days;
# A user-supplied instrumental recorded under uploads/ expires (bucket lifecycle);
# fall back to (and repoint the job at) its staged copy under jobs/{id}/.
if getattr(job, 'existing_instrumental_gcs_path', None):
job.existing_instrumental_gcs_path = resolve_existing_instrumental(
+1 -1
View File
@@ -557,7 +557,7 @@ class VideoWorkerOrchestrator:
if src != dest:
self.storage.copy_blob(src, dest)
self.job_log.info(f"Staged custom instrumental for GCE encoder: {dest}")
# uploads/ expires after 7 days: point the job at the
# uploads/ expires (bucket lifecycle): point the job at the
# kept job-root copy for future re-renders/regenerates.
if src.startswith("uploads/") and self.job_manager:
self.job_manager.update_job(
+2
View File
@@ -378,6 +378,8 @@ Downloads of archived files (`GET /api/jobs/{id}/download/{category}/{key}`) ret
Admin variant (optional quiet run, no rate limit): `POST /api/admin/jobs/{job_id}/regenerate` with `{"notify_customer": false}`.
**User inputs** (what regenerate/re-render needs) live in `jobs/{id}/input/**`: every pipeline worker copies `uploads/{id}/**` there on entry and repoints the job (`backend/services/input_persistence.py`; `uploads/` expires after 180 days). Backfill for jobs no worker will touch again: `POST /api/admin/persist-uploads` (admin; `?dry_run=false` to apply) → `{dry_run, jobs: {job_id: {copied, updated} | {would_copy} | {orphan: n} | {error}}}`; folders without a job document are reported as `orphan` and left in place.
A public → private visibility change on a track whose finals were archived regenerates them first (`after: "change_to_private"`), then runs the redistribution automatically.
### Review
+3 -2
View File
@@ -1,6 +1,6 @@
# Disaster Recovery Runbook
**Last updated:** 2026-10-01
**Last updated:** 2026-10-08
**Design doc:** `docs/archive/2026-03-27-business-continuity-design.md`
**Implementation plan:** `docs/archive/2026-03-29-business-continuity-plan.md`
**External services reference:** `docs/archive/2026-03-29-external-services-config.md`
@@ -13,7 +13,8 @@
| BigQuery (weekly) | S3 `bigquery/daily-refresh/` | Load Parquet files | 7 days |
| BigQuery (monthly) | S3 `bigquery/musicbrainz/` | Load Parquet files | 30 days |
| BigQuery (Spotify) | S3 `bigquery/spotify/` | Load from Glacier Deep Archive | N/A (static) |
| GCS job files | **GCS only** — object versioning (non-current kept 30d) + soft-delete (7d). Off-site S3 copy **stopped 2026-10-01**; S3 `gcs/job-files/` is a frozen snapshot up to 2026-09-30 | Restore non-current/soft-deleted generation in GCS | N/A off-site (frozen); in-GCS recovery window 7–30d |
| GCS job files | **GCS only** — object versioning (non-current kept 7d since 2026-10-03) + soft-delete (7d). Off-site S3 copy **stopped 2026-10-01**; S3 `gcs/job-files/` is a frozen snapshot up to 2026-09-30 | Restore non-current/soft-deleted generation in GCS | N/A off-site (frozen); in-GCS recovery window 7–30d |
| User-uploaded inputs (mix, own instrumental, lyrics, style uploads) | `jobs/{id}/input/**` (KEEP forever; copied from `uploads/{id}/` by every pipeline worker, `backend/services/input_persistence.py`). `uploads/` expires after **180 days** (was 7 until 2026-10-08) | Read from `jobs/{id}/input/`; backfill stragglers with `POST /api/admin/persist-uploads?dry_run=false` | Irreplaceable: no off-site copy. Lost for 64 Jan–Feb 2026 consumer jobs + BYO-instrumental jobs before the fix |
| Git repos (code) | S3 `git-repos/{owner}/{repo}.bundle` (+ `manifest.json`) | `git clone <bundle>` then push to a new remote | 7 days (weekly, Sundays; incremental) |
| Secret Manager | S3 `secrets/YYYY-MM-DD.bin` (sealed-box encrypted) | Decrypt with private key from KeepassXC | 24h |
| AWS credentials (function) | GCP Secret Manager `aws-backup-credentials` | Read directly | On-change |
+23
View File
@@ -6,6 +6,29 @@ Key insights for future AI agents working on this codebase.
---
## User uploads sat under a 7-day-expiring prefix — inputs are irreplaceable (Oct 2026, v0.268.2)
Every upload lands in `uploads/{job_id}/`, which had a bucket lifecycle **delete after 7 days**. Only
the audio worker (stem separation) copied the mix to `jobs/{id}/input/`, best-effort. Bring-your-own-
instrumental jobs (all tenant jobs) skip separation, the uploaded instrumental was only staged at
render time, and lyrics/style uploads were never copied — so a job left in review >7 days, or
re-rendered after 7 days, 404'd on its own input (randy-vild re-renders, 2026-10-08). Firestore also
showed 64 completed Jan–Feb 2026 consumer jobs whose mix is gone for good (no S3 copy: the off-site
sync only ever covered `jobs/`, `tenants/`, `themes/`).
- **Rule:** anything a user gave us is never under an expiring prefix once a job exists. Every
pipeline worker calls `input_persistence.ensure_job_inputs_persisted` on entry (server-side copy of
`uploads/{id}/**` → `jobs/{id}/input/**` + repoint every job field); `uploads/` now expires after
180 days as a pure staging area. `jobs/{id}/input/` is KEEP-forever in `storage_retention`.
- **Don't confine persistence to one stage**: inputs arrive late (conformed instrumentals, style
uploads) and stages get skipped per job type. Make it idempotent and call it everywhere.
- **Lifecycle rules are data-retention policy**: review them like deletes — a `matches_prefixes`
entry is a silent `rm -rf` on a timer.
- Recovery that worked: originals in Andrew's Dropbox, pulled GCP-side (Cloud Shell) via the
folder's existing shared link (the backend Dropbox app lacks `files.content.read`, but
`sharing.read` + `sharing_get_shared_link_file` works), conformed instrumentals re-derived
deterministically with `conform_instrumental` and checked against `state_data.instrumental_conformed`.
## Public-repo CI artifacts published the real test card and the admin token (Oct 2026)
Investigating E2E Daily #200 ("Stage 1 failed", Stripe self-heal "not needed") showed two things:
+1
View File
@@ -83,6 +83,7 @@
- Lyrics review Waveforms toggle: shows a spinner and tooltip while audio separation is still producing the vocal stem (the peaks endpoint returns 202). Previously the rows were just empty (`VocalsAudioDataLoaderContext.status`).
- **Translated lyrics: Japanese/Chinese translations no longer split mid-word** (2026-10-01, v0.260.1): when a long lyric line is split across rows, its translation is shared between them. Unspaced scripts were cut at an arbitrary character, so prod job ca2e881a showed 無|駄 ("waste") across two rows. `split_text_proportionally` now prefers punctuation, then a break after a Japanese particle or before a kanji/katakana word start, and only cuts mid-word as a last resort; it no longer leaves a piece empty when there are enough characters. Thai can still split inside a word (no word-boundary data).
- **Translated lyrics for language learners** (2026-10-01, v0.260.0): an "Add translated lyrics" option at job creation (any job, free for now) with a language picker (33 UI locales) and a live preview of the real video frame (`GET /api/themes/translation-preview`). After review, the render worker translates the final lyrics line by line with Gemini (`backend/services/lyrics_translation.py`, GCS-cached so re-renders are free) into `lyrics/translations.json`; the landscape and portrait videos draw each translation in smaller unsung-colour text beneath its line, 3 lines per screen (`ass/config.py:translation_layout`, `LyricsLine._create_translation_event`). Translations are their own ASS events with auto paragraph direction, so mixed scripts work (a Hebrew song keeps its right-to-left highlight with an English translation beneath, and vice versa); long translations wrap/scale to fit; a line split by the resizer shares its translation across the pieces. Not editable; a translation failure renders without it (`state_data.lyrics_translation`). See [archive/2026-10-01-translated-lyrics-plan.md](archive/2026-10-01-translated-lyrics-plan.md).
- **User-uploaded inputs kept forever** (2026-10-08, v0.268.2): `uploads/` had a 7-day lifecycle delete and only the audio worker copied the mix out, so bring-your-own-instrumental/tenant jobs (and any job left in review >7 days) lost their mix and instrumental; re-renders 404'd. Every pipeline worker now copies `uploads/{id}/**` to `jobs/{id}/input/**` and repoints the job (`backend/services/input_persistence.py`); `POST /api/admin/persist-uploads` backfills; `uploads/` now expires after 180 days. randy-vild inputs were restored from Dropbox. See [LESSONS-LEARNED.md](LESSONS-LEARNED.md).
- **End screen fully editable + previewed; title card / end screen can be left out** (2026-10-08, v0.268.0): Randy Vild reported "large text at the end even though I have in my settings smaller text like in my beginning". The theme editor's sizes/positions only edited `intro.*`; the end screen keeps its own `end.*` layout (title/artist regions, artist colour, capitals), and the editor neither showed it nor previewed it. The End screen tab now has its own colours, size/position sliders, show/hide for the song title and artist name, capitals toggles, and **Copy title card layout** (matches the title card and moves the closing message above the title so nothing overlaps). The preview renders the real end screen (`create_end_video`). Both tabs have an "Include …" switch (`intro.enabled` / `end.enabled`); leaving a screen out flows through the screens worker, `/retry`, the orchestrator, the GCE encoder, `LocalEncodingService` and the portrait render (see [API.md](API.md#tenant-theme-editor-tenant-users)).
- **Theme edits reach tracks in progress; outdated finished tracks get a re-render prompt** (2026-10-06, v0.267.0): Randy Vild edited his theme and the next video still had the old title card. Its job had copied the theme at creation, 30 minutes before the edit, and was waiting for encoding capacity when he saved. A theme save now re-applies the theme to every tenant track that hasn't started rendering. If a track's screens were already made, the render worker regenerates them first, at render time, so nobody is interrupted mid-review. Editing a finished tenant track also picks up the current theme. After saving, the theme editor lists finished tracks that still have the old look and offers a quiet **Re-render all N** (no per-track emails). Those tracks' cards show an "Older theme" badge. Admins no longer see two identical Re-render buttons on portal tracks (`backend/services/theme_change_service.py`).
- **Re-render works for tenants that archive to Dropbox** (2026-09-30, v0.254.1): randy-vild jobs get a brand code and a Dropbox archive folder (`RVILD-0001 - …`), so v0.253.0's "never distributed" guard rejected the very jobs the button was for. A re-render now reuses the job's brand code and overwrites the same folder instead of allocating a new code. YouTube/GDrive-published jobs are still rejected.
+15 -1
View File
@@ -65,7 +65,21 @@ def create_bucket() -> storage.Bucket:
action=storage.BucketLifecycleRuleActionArgs(type="Delete"),
condition=storage.BucketLifecycleRuleConditionArgs(
age=7,
matches_prefixes=["temp/", "uploads/"]
matches_prefixes=["temp/"]
),
),
# uploads/ holds USER INPUTS (mix, own instrumental, lyrics, style
# uploads) — irreplaceable. Workers copy them to jobs/{id}/input/
# (kept forever; backend/services/input_persistence.py), so uploads/
# is only a staging area — but it used to expire after 7 days, and
# jobs that skipped the copy (bring-your-own-instrumental / tenant
# jobs, anything left in review >7 days) lost their inputs for good
# (2026-10-08). 6 months leaves ample time for any worker to persist.
storage.BucketLifecycleRuleArgs(
action=storage.BucketLifecycleRuleActionArgs(type="Delete"),
condition=storage.BucketLifecycleRuleConditionArgs(
age=180,
matches_prefixes=["uploads/"]
),
),
# Noncurrent (overwritten OR deleted) versions are kept 7 days (was