mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
## Summary Two invisible upload failures on the same song led to this: - **Job 8729f19a**: large audio upload (signed-URL flow) abandoned mid-upload. The only UI signal was the "Creating..." button text and the card already said "Setting up", so the job sat `pending` forever with the credit spent. - **Job 2d0c1ae0**: 44 MB instrumental upload on the review screen was a multipart POST through Cloud Run (32 MiB cap) and failed silently. Only the CORS preflight was logged. ## Changes **Shared upload layer (frontend)**: all user uploads now go through one path: - `lib/upload.ts`: signed-URL PUT with progress, sequential multi-file upload with aggregate progress, client-side duration check. - `hooks/useUploadTask.ts`: progress state plus a `beforeunload` guard. - `components/upload/UploadProgressModal.tsx`: a modal the user can't close, showing progress, speed, ETA, "file N of M" and a keep-tab-open warning. - Used by guided job creation, tenant single job and instrumental review. Tenant bulk keeps its per-row table and uses the shared guard. **Guided flow** - Always uses the signed-URL path, which also fixes `requires_audio_edit` being silently dropped for uploads of 25MB or more. - Private upload jobs can attach their own instrumental (`existing_instrumental`). This skips separation and instrumental review. The duration is checked before upload, and the backing-vocals option is hidden. **Instrumental review** - New `instrumental-upload-url` and `instrumental-upload-complete` endpoints: direct-to-GCS upload, then a finalize step. - One shared processing helper with the multipart route: duration check, FLAC conversion, `stems.custom_instrumental`. - Returns a signed `audio_url`. The old post-upload preview URL had no backend route. - Inline error plus toast for mismatches and failures. **Backend robustness** - `state_data.awaiting_upload`: the dashboard shows "Waiting for upload" instead of "Setting up". - The hourly stale-review run also cancels and refunds non-tenant jobs still awaiting upload after 2h with nothing uploaded. - `uploads-complete` now checks ownership (previously any signed-in user could finalize any pending job). On an instrumental duration mismatch it cancels and refunds the job instead of stranding it. ## Testing - `make test`: backend 1839 + 4788 + 150 emulator passed; frontend Jest 1391 passed across 117 suites. - New tests: - backend: stale-upload processor; signed instrumental endpoints (15 cases); `requires_audio_edit` persistence; uploads-complete ownership and cancel-on-mismatch - frontend: upload lib, `useUploadTask`, modal (speed/ETA, can't be dismissed), `OwnInstrumentalField`, `TenantJobFlow`, API flows - Manual check against the real frontend (local dev, prod read-only; create, upload and finalize endpoints mocked, uploads sent to a throttled local PUT server, so no prod data was changed): - guided flow with and without an instrumental - instrumental review upload on 2d0c1ae0, including a mismatch caught before upload - Escape and outside clicks don't close the modal; navigating away triggers the `beforeunload` prompt Plan: `docs/archive/2026-09-26-unified-upload-flow-plan.md` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
106 lines
4.0 KiB
Python
106 lines
4.0 KiB
Python
"""
|
|
Stale upload processor.
|
|
|
|
The signed-URL upload flow creates the job (and charges the credit) BEFORE the
|
|
browser PUTs the audio straight to GCS; the job stays PENDING with
|
|
``state_data.awaiting_upload`` until the client calls ``uploads-complete``.
|
|
If the tab is closed or the upload fails, nothing ever advances the job — it
|
|
sits at "Waiting for upload" forever with the credit spent.
|
|
|
|
This sweep cancels such jobs once their signed upload URLs have long expired
|
|
(the URLs are valid for 60 min), which refunds the credit via ``cancel_job``.
|
|
|
|
Called from the hourly stale-review scheduler endpoint.
|
|
"""
|
|
import logging
|
|
from datetime import datetime, timezone
|
|
from typing import Any, Dict
|
|
|
|
from backend.models.job import JobStatus
|
|
from backend.services.firestore_service import FirestoreService
|
|
from backend.services.job_manager import JobManager
|
|
from backend.services.storage_service import StorageService
|
|
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
# Signed upload URLs expire after 60 minutes, but GCS only checks expiry when the
|
|
# PUT starts — a big file on a slow uplink can still be streaming hours later, and
|
|
# a single-shot PUT isn't visible until it finishes. 6h leaves plenty of margin.
|
|
STALE_UPLOAD_HOURS = 6
|
|
# Some but not all files landed (e.g. mix uploaded, instrumental didn't) and
|
|
# uploads-complete was never called: give a human a day, then cancel + refund.
|
|
STALE_PARTIAL_UPLOAD_HOURS = 24
|
|
|
|
CANCEL_REASON = "Audio upload never finished. Please submit the song again."
|
|
|
|
|
|
def process_stale_uploads() -> Dict[str, Any]:
|
|
"""
|
|
Cancel (and refund) PENDING jobs whose browser upload never completed.
|
|
|
|
Skips tenant bulk jobs (``state_data.batch_id``): they use resumable sessions
|
|
that can be resumed for days via the re-pick recovery flow.
|
|
Jobs with some objects under ``uploads/{job_id}/`` get a longer grace period
|
|
(``STALE_PARTIAL_UPLOAD_HOURS``) before being cancelled.
|
|
"""
|
|
firestore = FirestoreService()
|
|
job_manager = JobManager()
|
|
storage = StorageService()
|
|
|
|
cancelled = 0
|
|
skipped_has_files = 0
|
|
errors = []
|
|
|
|
try:
|
|
pending_jobs = firestore.list_jobs(status=JobStatus.PENDING, limit=500)
|
|
except Exception as e:
|
|
logger.error(f"Failed to query pending jobs: {e}")
|
|
return {"status": "error", "cancelled": 0, "errors": [str(e)]}
|
|
|
|
now = datetime.now(timezone.utc)
|
|
|
|
for job in pending_jobs:
|
|
try:
|
|
state_data = job.state_data or {}
|
|
if not state_data.get('awaiting_upload'):
|
|
continue
|
|
if state_data.get('batch_id'):
|
|
continue
|
|
|
|
created_at = job.created_at
|
|
if created_at.tzinfo is None:
|
|
created_at = created_at.replace(tzinfo=timezone.utc)
|
|
hours_elapsed = (now - created_at).total_seconds() / 3600
|
|
if hours_elapsed < STALE_UPLOAD_HOURS:
|
|
continue
|
|
|
|
if hours_elapsed < STALE_PARTIAL_UPLOAD_HOURS and storage.list_files(f"uploads/{job.job_id}/"):
|
|
skipped_has_files += 1
|
|
logger.warning(
|
|
f"Job {job.job_id}: awaiting_upload for {hours_elapsed:.1f}h with some files "
|
|
f"under uploads/ — uploads-complete never called; cancelling at "
|
|
f"{STALE_PARTIAL_UPLOAD_HOURS}h"
|
|
)
|
|
continue
|
|
|
|
logger.info(
|
|
f"Job {job.job_id}: upload never completed after {hours_elapsed:.1f}h, "
|
|
f"cancelling with refund"
|
|
)
|
|
if job_manager.cancel_job(job.job_id, reason=CANCEL_REASON):
|
|
cancelled += 1
|
|
else:
|
|
logger.warning(f"Job {job.job_id}: cancel_job returned False")
|
|
except Exception as e:
|
|
error_msg = f"Job {getattr(job, 'job_id', '?')}: stale upload processing failed: {e}"
|
|
logger.error(error_msg)
|
|
errors.append(error_msg)
|
|
|
|
return {
|
|
"status": "completed",
|
|
"cancelled": cancelled,
|
|
"skipped_has_files": skipped_has_files,
|
|
"errors": errors,
|
|
}
|