Files
karaoke-gen/backend/workers/stale_upload_processor.py
Andrew Beveridge ☄️andClaude Opus 5.5 455fb11a14 feat(upload): unified browser upload flow, bring-your-own instrumental, stranded-upload cleanup (v0.243.0) (#1058)
## Summary
Two invisible upload failures on the same song led to this:
- **Job 8729f19a**: large audio upload (signed-URL flow) abandoned
mid-upload. The only UI signal was the "Creating..." button text and the
card already said "Setting up", so the job sat `pending` forever with
the credit spent.
- **Job 2d0c1ae0**: 44 MB instrumental upload on the review screen was a
multipart POST through Cloud Run (32 MiB cap) and failed silently. Only
the CORS preflight was logged.

## Changes
**Shared upload layer (frontend)**: all user uploads now go through one
path:
- `lib/upload.ts`: signed-URL PUT with progress, sequential multi-file
upload with aggregate progress, client-side duration check.
- `hooks/useUploadTask.ts`: progress state plus a `beforeunload` guard.
- `components/upload/UploadProgressModal.tsx`: a modal the user can't
close, showing progress, speed, ETA, "file N of M" and a keep-tab-open
warning.
- Used by guided job creation, tenant single job and instrumental
review. Tenant bulk keeps its per-row table and uses the shared guard.

**Guided flow**
- Always uses the signed-URL path, which also fixes
`requires_audio_edit` being silently dropped for uploads of 25MB or
more.
- Private upload jobs can attach their own instrumental
(`existing_instrumental`). This skips separation and instrumental
review. The duration is checked before upload, and the backing-vocals
option is hidden.

**Instrumental review**
- New `instrumental-upload-url` and `instrumental-upload-complete`
endpoints: direct-to-GCS upload, then a finalize step.
- One shared processing helper with the multipart route: duration check,
FLAC conversion, `stems.custom_instrumental`.
- Returns a signed `audio_url`. The old post-upload preview URL had no
backend route.
- Inline error plus toast for mismatches and failures.

**Backend robustness**
- `state_data.awaiting_upload`: the dashboard shows "Waiting for upload"
instead of "Setting up".
- The hourly stale-review run also cancels and refunds non-tenant jobs
still awaiting upload after 2h with nothing uploaded.
- `uploads-complete` now checks ownership (previously any signed-in user
could finalize any pending job). On an instrumental duration mismatch it
cancels and refunds the job instead of stranding it.

## Testing
- `make test`: backend 1839 + 4788 + 150 emulator passed; frontend Jest
1391 passed across 117 suites.
- New tests:
- backend: stale-upload processor; signed instrumental endpoints (15
cases); `requires_audio_edit` persistence; uploads-complete ownership
and cancel-on-mismatch
- frontend: upload lib, `useUploadTask`, modal (speed/ETA, can't be
dismissed), `OwnInstrumentalField`, `TenantJobFlow`, API flows
- Manual check against the real frontend (local dev, prod read-only;
create, upload and finalize endpoints mocked, uploads sent to a
throttled local PUT server, so no prod data was changed):
  - guided flow with and without an instrumental
- instrumental review upload on 2d0c1ae0, including a mismatch caught
before upload
- Escape and outside clicks don't close the modal; navigating away
triggers the `beforeunload` prompt

Plan: `docs/archive/2026-09-26-unified-upload-flow-plan.md`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-26 20:25:00 -04:00

106 lines
4.0 KiB
Python

"""
Stale upload processor.
The signed-URL upload flow creates the job (and charges the credit) BEFORE the
browser PUTs the audio straight to GCS; the job stays PENDING with
``state_data.awaiting_upload`` until the client calls ``uploads-complete``.
If the tab is closed or the upload fails, nothing ever advances the job — it
sits at "Waiting for upload" forever with the credit spent.
This sweep cancels such jobs once their signed upload URLs have long expired
(the URLs are valid for 60 min), which refunds the credit via ``cancel_job``.
Called from the hourly stale-review scheduler endpoint.
"""
import logging
from datetime import datetime, timezone
from typing import Any, Dict
from backend.models.job import JobStatus
from backend.services.firestore_service import FirestoreService
from backend.services.job_manager import JobManager
from backend.services.storage_service import StorageService
logger = logging.getLogger(__name__)
# Signed upload URLs expire after 60 minutes, but GCS only checks expiry when the
# PUT starts — a big file on a slow uplink can still be streaming hours later, and
# a single-shot PUT isn't visible until it finishes. 6h leaves plenty of margin.
STALE_UPLOAD_HOURS = 6
# Some but not all files landed (e.g. mix uploaded, instrumental didn't) and
# uploads-complete was never called: give a human a day, then cancel + refund.
STALE_PARTIAL_UPLOAD_HOURS = 24
CANCEL_REASON = "Audio upload never finished. Please submit the song again."
def process_stale_uploads() -> Dict[str, Any]:
"""
Cancel (and refund) PENDING jobs whose browser upload never completed.
Skips tenant bulk jobs (``state_data.batch_id``): they use resumable sessions
that can be resumed for days via the re-pick recovery flow.
Jobs with some objects under ``uploads/{job_id}/`` get a longer grace period
(``STALE_PARTIAL_UPLOAD_HOURS``) before being cancelled.
"""
firestore = FirestoreService()
job_manager = JobManager()
storage = StorageService()
cancelled = 0
skipped_has_files = 0
errors = []
try:
pending_jobs = firestore.list_jobs(status=JobStatus.PENDING, limit=500)
except Exception as e:
logger.error(f"Failed to query pending jobs: {e}")
return {"status": "error", "cancelled": 0, "errors": [str(e)]}
now = datetime.now(timezone.utc)
for job in pending_jobs:
try:
state_data = job.state_data or {}
if not state_data.get('awaiting_upload'):
continue
if state_data.get('batch_id'):
continue
created_at = job.created_at
if created_at.tzinfo is None:
created_at = created_at.replace(tzinfo=timezone.utc)
hours_elapsed = (now - created_at).total_seconds() / 3600
if hours_elapsed < STALE_UPLOAD_HOURS:
continue
if hours_elapsed < STALE_PARTIAL_UPLOAD_HOURS and storage.list_files(f"uploads/{job.job_id}/"):
skipped_has_files += 1
logger.warning(
f"Job {job.job_id}: awaiting_upload for {hours_elapsed:.1f}h with some files "
f"under uploads/ — uploads-complete never called; cancelling at "
f"{STALE_PARTIAL_UPLOAD_HOURS}h"
)
continue
logger.info(
f"Job {job.job_id}: upload never completed after {hours_elapsed:.1f}h, "
f"cancelling with refund"
)
if job_manager.cancel_job(job.job_id, reason=CANCEL_REASON):
cancelled += 1
else:
logger.warning(f"Job {job.job_id}: cancel_job returned False")
except Exception as e:
error_msg = f"Job {getattr(job, 'job_id', '?')}: stale upload processing failed: {e}"
logger.error(error_msg)
errors.append(error_msg)
return {
"status": "completed",
"cancelled": cancelled,
"skipped_has_files": skipped_has_files,
"errors": errors,
}