mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary
`uploads/` had a bucket lifecycle **delete after 7 days**, and user
inputs were only partly copied out of it. The only copy step was the
audio worker (stem separation) copying the mix to `jobs/{id}/input/`,
best-effort. Everything else stayed in `uploads/`:
- bring-your-own-instrumental / tenant jobs, which skip separation;
- the uploaded instrumental, which was only staged at render time;
- lyrics and style uploads, which were never copied.
Result:
- randy-vild re-renders 404'd on their own input (2026-10-08). That data
was restored from Andrew's Dropbox.
- Firestore shows 64 completed Jan–Feb 2026 consumer jobs, plus
BYO-instrumental jobs, whose inputs are gone permanently.
## Changes
- **`backend/services/input_persistence.py`** (new):
- Server-side copies `uploads/{id}/**` to `jobs/{id}/input/**` and
repoints every job field that referenced the upload:
`input_media_gcs_path`, `existing_instrumental_gcs_path`,
`lyrics_file_gcs_path`, `style_params_gcs_path`, plus any matching value
under `file_urls` / `state_data` / `style_assets`.
- Idempotent: unchanged copies are skipped.
- The audio, lyrics, screens, render-video and video workers all call it
on entry, via `asyncio.to_thread`. On failure it logs an ERROR, which
alerts, but doesn't fail the job.
- **`POST /api/admin/persist-uploads`** (dry run by default): backfill
for jobs no worker will touch again. Lookup failures are isolated per
job.
- **GCE encoder:** prefers the explicitly downloaded top-level
`existing_instrumental.*`. The recursive glob could otherwise return the
raw, unconformed copy now kept under `input/audio/`.
- **Pulumi:** `uploads/` lifecycle goes from 7 to **180 days**; `temp/`
stays at 7. **Already applied** and verified directly on the bucket.
- **Docs:** LESSONS-LEARNED, DISASTER-RECOVERY (new user-inputs row; the
stale "30d noncurrent" figure corrected to 7d), API, README.
## Verification
- `make test` all green.
- 16 new tests:
- copy mapping, repointing, idempotency, re-copy when a kept copy
differs, already-expired paths left alone;
- the worker wrapper never raises;
- an AST guard that every pipeline worker calls the copy step via
`to_thread`;
- backfill dry run / apply / error isolation;
- the encoder's instrumental preference.
- Local CodeRabbit: 2 findings, both fixed (lyrics path, per-job
lookup); re-review clean.
## After merge
Run `POST /api/admin/persist-uploads` (dry run, then `?dry_run=false`)
to copy the inputs of the 51 jobs that still have objects in `uploads/`.
@coderabbitai ignore
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
103 lines
4.8 KiB
Python
103 lines
4.8 KiB
Python
"""
|
|
Cloud Storage resources.
|
|
|
|
Manages the GCS bucket for uploads, outputs, and temporary files.
|
|
"""
|
|
|
|
import pulumi
|
|
import pulumi_gcp as gcp
|
|
from pulumi_gcp import storage
|
|
|
|
from config import PROJECT_ID
|
|
|
|
|
|
def create_bucket() -> storage.Bucket:
|
|
"""
|
|
Create the main GCS bucket with lifecycle rules.
|
|
|
|
Returns:
|
|
storage.Bucket: The created bucket resource.
|
|
"""
|
|
bucket = storage.Bucket(
|
|
"karaoke-storage",
|
|
name=f"karaoke-gen-storage-{PROJECT_ID}",
|
|
location="US-CENTRAL1",
|
|
force_destroy=False, # Prevent accidental deletion
|
|
uniform_bucket_level_access=True,
|
|
# Never publicly readable: every consumer is an authenticated service
|
|
# account and browsers only ever get V4 signed URLs. (An out-of-band
|
|
# `allUsers` objectViewer grant existed 2025-12-22 → 2026-09-28 to host a
|
|
# debug page; it exposed every customer file AND made GCS edge-cache
|
|
# objects for 1h, so backend reads of overwritten configs went stale.)
|
|
public_access_prevention="enforced",
|
|
cors=[
|
|
storage.BucketCorArgs(
|
|
# Any origin: tenant portals get new *.nomadkaraoke.com
|
|
# subdomains at runtime (admin tenant console), and a per-tenant
|
|
# list here can't keep up. Safe because the bucket is private —
|
|
# a cross-origin request only succeeds with a signed URL or an
|
|
# origin-bound resumable session, which are the auth.
|
|
origins=["*"],
|
|
# PUT: direct-to-GCS uploads. GET/HEAD: the lyrics-review vocals
|
|
# waveform fetch()es the signed stem URL and reads its bytes via
|
|
# decodeAudioData — unlike <audio> playback, that's a cross-origin
|
|
# read and needs CORS. Without GET the browser blocks it with
|
|
# "Access-Control-Allow-Origin missing".
|
|
methods=["GET", "HEAD", "PUT"],
|
|
response_headers=["Content-Type"],
|
|
max_age_seconds=3600,
|
|
),
|
|
],
|
|
versioning=storage.BucketVersioningArgs(enabled=True),
|
|
# Autoclass (2026-09-26 GCP cost cut): objects not read for 30 days move
|
|
# to Nearline automatically and back to Standard on the next read, with
|
|
# no retrieval or early-deletion fees (operations bill at Standard
|
|
# rates). Terminal class NEARLINE (not ARCHIVE) because customers
|
|
# re-download old job outputs. Incompatible with lifecycle
|
|
# SetStorageClass rules — only Delete rules may live below.
|
|
autoclass=storage.BucketAutoclassArgs(
|
|
enabled=True,
|
|
terminal_storage_class="NEARLINE",
|
|
),
|
|
soft_delete_policy=storage.BucketSoftDeletePolicyArgs(retention_duration_seconds=604800),
|
|
lifecycle_rules=[
|
|
storage.BucketLifecycleRuleArgs(
|
|
action=storage.BucketLifecycleRuleActionArgs(type="Delete"),
|
|
condition=storage.BucketLifecycleRuleConditionArgs(
|
|
age=7,
|
|
matches_prefixes=["temp/"]
|
|
),
|
|
),
|
|
# uploads/ holds USER INPUTS (mix, own instrumental, lyrics, style
|
|
# uploads) — irreplaceable. Workers copy them to jobs/{id}/input/
|
|
# (kept forever; backend/services/input_persistence.py), so uploads/
|
|
# is only a staging area — but it used to expire after 7 days, and
|
|
# jobs that skipped the copy (bring-your-own-instrumental / tenant
|
|
# jobs, anything left in review >7 days) lost their inputs for good
|
|
# (2026-10-08). 6 months leaves ample time for any worker to persist.
|
|
storage.BucketLifecycleRuleArgs(
|
|
action=storage.BucketLifecycleRuleActionArgs(type="Delete"),
|
|
condition=storage.BucketLifecycleRuleConditionArgs(
|
|
age=180,
|
|
matches_prefixes=["uploads/"]
|
|
),
|
|
),
|
|
# Noncurrent (overwritten OR deleted) versions are kept 7 days (was
|
|
# 30 until the 2026-10-03 storage-retention work): long enough to undo
|
|
# a bad overwrite/purge, short enough not to double-bill.
|
|
# No num_newer_versions condition: a DELETED object's last version has
|
|
# no newer version, so the old `num_newer_versions=1` rule kept every
|
|
# deleted file's bytes forever (and would have made the storage-
|
|
# retention purge free nothing).
|
|
storage.BucketLifecycleRuleArgs(
|
|
action=storage.BucketLifecycleRuleActionArgs(type="Delete"),
|
|
condition=storage.BucketLifecycleRuleConditionArgs(
|
|
with_state="ARCHIVED",
|
|
days_since_noncurrent_time=7,
|
|
),
|
|
),
|
|
],
|
|
)
|
|
|
|
return bucket
|