Files
Andrew Beveridge ☄️andClaude Opus 5.5 983f6ecdcd fix(storage): keep user-uploaded inputs forever; uploads/ expires after 180d not 7d (v0.268.2) (#1137)
## Summary
`uploads/` had a bucket lifecycle **delete after 7 days**, and user
inputs were only partly copied out of it. The only copy step was the
audio worker (stem separation) copying the mix to `jobs/{id}/input/`,
best-effort. Everything else stayed in `uploads/`:
- bring-your-own-instrumental / tenant jobs, which skip separation;
- the uploaded instrumental, which was only staged at render time;
- lyrics and style uploads, which were never copied.

Result:
- randy-vild re-renders 404'd on their own input (2026-10-08). That data
was restored from Andrew's Dropbox.
- Firestore shows 64 completed Jan–Feb 2026 consumer jobs, plus
BYO-instrumental jobs, whose inputs are gone permanently.

## Changes
- **`backend/services/input_persistence.py`** (new):
- Server-side copies `uploads/{id}/**` to `jobs/{id}/input/**` and
repoints every job field that referenced the upload:
`input_media_gcs_path`, `existing_instrumental_gcs_path`,
`lyrics_file_gcs_path`, `style_params_gcs_path`, plus any matching value
under `file_urls` / `state_data` / `style_assets`.
  - Idempotent: unchanged copies are skipped.
- The audio, lyrics, screens, render-video and video workers all call it
on entry, via `asyncio.to_thread`. On failure it logs an ERROR, which
alerts, but doesn't fail the job.
- **`POST /api/admin/persist-uploads`** (dry run by default): backfill
for jobs no worker will touch again. Lookup failures are isolated per
job.
- **GCE encoder:** prefers the explicitly downloaded top-level
`existing_instrumental.*`. The recursive glob could otherwise return the
raw, unconformed copy now kept under `input/audio/`.
- **Pulumi:** `uploads/` lifecycle goes from 7 to **180 days**; `temp/`
stays at 7. **Already applied** and verified directly on the bucket.
- **Docs:** LESSONS-LEARNED, DISASTER-RECOVERY (new user-inputs row; the
stale "30d noncurrent" figure corrected to 7d), API, README.

## Verification
- `make test` all green.
- 16 new tests:
- copy mapping, repointing, idempotency, re-copy when a kept copy
differs, already-expired paths left alone;
  - the worker wrapper never raises;
- an AST guard that every pipeline worker calls the copy step via
`to_thread`;
  - backfill dry run / apply / error isolation;
  - the encoder's instrumental preference.
- Local CodeRabbit: 2 findings, both fixed (lyrics path, per-job
lookup); re-review clean.

## After merge
Run `POST /api/admin/persist-uploads` (dry run, then `?dry_run=false`)
to copy the inputs of the 51 jobs that still have objects in `uploads/`.

@coderabbitai ignore

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 17:33:06 -04:00

103 lines
4.8 KiB
Python

"""
Cloud Storage resources.
Manages the GCS bucket for uploads, outputs, and temporary files.
"""
import pulumi
import pulumi_gcp as gcp
from pulumi_gcp import storage
from config import PROJECT_ID
def create_bucket() -> storage.Bucket:
"""
Create the main GCS bucket with lifecycle rules.
Returns:
storage.Bucket: The created bucket resource.
"""
bucket = storage.Bucket(
"karaoke-storage",
name=f"karaoke-gen-storage-{PROJECT_ID}",
location="US-CENTRAL1",
force_destroy=False, # Prevent accidental deletion
uniform_bucket_level_access=True,
# Never publicly readable: every consumer is an authenticated service
# account and browsers only ever get V4 signed URLs. (An out-of-band
# `allUsers` objectViewer grant existed 2025-12-22 → 2026-09-28 to host a
# debug page; it exposed every customer file AND made GCS edge-cache
# objects for 1h, so backend reads of overwritten configs went stale.)
public_access_prevention="enforced",
cors=[
storage.BucketCorArgs(
# Any origin: tenant portals get new *.nomadkaraoke.com
# subdomains at runtime (admin tenant console), and a per-tenant
# list here can't keep up. Safe because the bucket is private —
# a cross-origin request only succeeds with a signed URL or an
# origin-bound resumable session, which are the auth.
origins=["*"],
# PUT: direct-to-GCS uploads. GET/HEAD: the lyrics-review vocals
# waveform fetch()es the signed stem URL and reads its bytes via
# decodeAudioData — unlike <audio> playback, that's a cross-origin
# read and needs CORS. Without GET the browser blocks it with
# "Access-Control-Allow-Origin missing".
methods=["GET", "HEAD", "PUT"],
response_headers=["Content-Type"],
max_age_seconds=3600,
),
],
versioning=storage.BucketVersioningArgs(enabled=True),
# Autoclass (2026-09-26 GCP cost cut): objects not read for 30 days move
# to Nearline automatically and back to Standard on the next read, with
# no retrieval or early-deletion fees (operations bill at Standard
# rates). Terminal class NEARLINE (not ARCHIVE) because customers
# re-download old job outputs. Incompatible with lifecycle
# SetStorageClass rules — only Delete rules may live below.
autoclass=storage.BucketAutoclassArgs(
enabled=True,
terminal_storage_class="NEARLINE",
),
soft_delete_policy=storage.BucketSoftDeletePolicyArgs(retention_duration_seconds=604800),
lifecycle_rules=[
storage.BucketLifecycleRuleArgs(
action=storage.BucketLifecycleRuleActionArgs(type="Delete"),
condition=storage.BucketLifecycleRuleConditionArgs(
age=7,
matches_prefixes=["temp/"]
),
),
# uploads/ holds USER INPUTS (mix, own instrumental, lyrics, style
# uploads) — irreplaceable. Workers copy them to jobs/{id}/input/
# (kept forever; backend/services/input_persistence.py), so uploads/
# is only a staging area — but it used to expire after 7 days, and
# jobs that skipped the copy (bring-your-own-instrumental / tenant
# jobs, anything left in review >7 days) lost their inputs for good
# (2026-10-08). 6 months leaves ample time for any worker to persist.
storage.BucketLifecycleRuleArgs(
action=storage.BucketLifecycleRuleActionArgs(type="Delete"),
condition=storage.BucketLifecycleRuleConditionArgs(
age=180,
matches_prefixes=["uploads/"]
),
),
# Noncurrent (overwritten OR deleted) versions are kept 7 days (was
# 30 until the 2026-10-03 storage-retention work): long enough to undo
# a bad overwrite/purge, short enough not to double-bill.
# No num_newer_versions condition: a DELETED object's last version has
# no newer version, so the old `num_newer_versions=1` rule kept every
# deleted file's bytes forever (and would have made the storage-
# retention purge free nothing).
storage.BucketLifecycleRuleArgs(
action=storage.BucketLifecycleRuleActionArgs(type="Delete"),
condition=storage.BucketLifecycleRuleConditionArgs(
with_state="ARCHIVED",
days_since_noncurrent_time=7,
),
),
],
)
return bucket