feat: precomputed vocals peaks — Waveforms strips paint sub-second (#1031)

## Summary

Final piece of the review-page speed work (#1024 → #1028 → #1029): the
Waveforms-mode strips no longer require downloading + WebAudio-decoding
the entire vocals stem (~7.5MB, ~4–6s) — the backend now serves a
**precomputed 400-peaks/s envelope as ~150KB of uint8/base64 JSON**,
cached in GCS and pre-warmed at prep time, so strips paint sub-second.

## How

- **`WaveformGenerator.generate_peaks()`** — numpy max-abs envelope,
bucket-for-bucket identical to the frontend's `computePeaks` (same
drift-free fractional-stride bucketing; exact match vs a naive reference
in tests).
- **`AudioAnalysisService.get_vocals_peaks()`** — GCS cache at
`review-audio/vocals_peaks_400.json`, validated against the stem path;
miss decodes the small transcoded OGG; upload best-effort.
- **`GET /api/review/{job_id}/vocals-peaks`** — off the event loop under
the shared waveform semaphore; `202 Retry-After: 15` while separation
runs (mirrors `/audio/vocals`); `Cache-Control: private, max-age=600`;
`peaks_per_second` clamped 50–1000.
- **`screens_worker`** pre-computes the envelope next to the existing
OGG + waveform pre-warm, so it's a pure cache read by the time a job
reaches review.
- **Frontend** — `VocalsAudioDataLoader` prefers the peaks endpoint (one
quick retry on transient errors) and **falls back to the original full
download+decode** on any terminal peaks failure, so a peaks-side problem
can never cost a user their waveforms. Local mode (no peaks URL) uses
the old path unchanged. Vocal stem selection factored into
`backend/utils/stems.py` (shared by routes + worker).

## Testing

- Package: 4 tests incl. exact naive-reference parity (**106 passed**)
- Backend: 4 service + 4 endpoint tests (**4518 passed**)
- Frontend: `fetchVocalsPeaks` units + 5 loader-behavior tests (prefers
peaks / falls back / one-retry-then-fallback / 202 polling / audio-only
mode) (**1307 passed**)
- `review-fast-load.spec.ts` prod E2E extended with a cached
vocals-peaks latency/shape check
- Post-deploy: will verify the endpoint on a real in-review job and
confirm payload size + latency

## Notes

- Existing in-review jobs have no cache yet — their first Waveforms open
computes + caches lazily (bounded); every open after that is sub-second.
New jobs are pre-warmed.
- 8-bit amplitude quantization is visually lossless at strip heights
(14px).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Andrew Beveridge ☄️andClaude Fable 5 authored and GitHub committed 2026-09-20 13:08:19 -04:00
1 parent 7cb5de729a
commit d85b6e271c
18 files changed
+744 -28

No files matched your search

+63 -13
View File
@@ -25,7 +25,7 @@ from pathlib import Path
from typing import Dict, Any, List, Literal, Optional, Set, Tuple
from fastapi import APIRouter, HTTPException, Request, Depends, Form, File, UploadFile
from fastapi.responses import RedirectResponse, Response
from fastapi.responses import JSONResponse, RedirectResponse, Response
from google.cloud.exceptions import NotFound
from google.cloud.firestore_v1.field_path import FieldPath
@@ -38,6 +38,7 @@ from backend.services.job_logging import job_log_context, JobLogger
from backend.services.tracing import create_span, add_span_attribute, add_span_event
from backend.services.encoding_service import get_encoding_service
from backend.api.dependencies import require_auth, require_review_auth
from backend.utils.stems import vocals_stem_path as _vocals_stem_path
from backend.services.auth_service import UserType
from backend.config import get_settings
from backend.i18n import t, get_locale_from_request
@@ -874,18 +875,7 @@ async def _stream_audio(job_id: str, request: Optional[Request] = None, stem: Li
if stem == "input":
audio_gcs_path = job.input_media_gcs_path
elif stem == "vocals":
stems = job.file_urls.get("stems", {}) if job.file_urls else {}
# Stem keys vary by separation model. We want the full vocal mix
# (lead + backing) so every word can be lined up against the waveform:
# - "vocals" : full vocal from a 2-stem split (rare)
# - "vocals_clean" : full vocal from the primary vocal/instrumental
# split (present on essentially all cloud jobs)
# - "lead_vocals" : last-resort fallback (misses backing lines)
# Use .get() so a missing key yields a clean 404 below, not a KeyError/500.
for key in ("vocals", "vocals_clean", "lead_vocals"):
if stems.get(key):
audio_gcs_path = stems[key]
break
audio_gcs_path = _vocals_stem_path(job)
if not audio_gcs_path:
# Audio separation is decoupled from the lyrics-review critical path, so a
# user can open review before the vocal stems have been uploaded (the stem
@@ -2201,6 +2191,66 @@ _WAVEFORM_MAX_CONCURRENCY = max(1, int(os.getenv("REVIEW_WAVEFORM_MAX_CONCURRENC
_waveform_semaphore = asyncio.Semaphore(_WAVEFORM_MAX_CONCURRENCY)
@router.get("/{job_id}/vocals-peaks")
async def get_vocals_peaks(
job_id: str,
peaks_per_second: int = 400,
auth_info: Tuple[str, str] = Depends(require_review_auth)
):
"""
Pre-computed peak envelope of the vocal stem (Waveforms review mode).
Replaces the frontend's download-and-decode of the whole vocals OGG
(~7 MB + WebAudio decode) with a ~150 KB JSON, so the per-segment
waveform strips paint sub-second on page load. Served from a persistent
GCS cache, pre-computed by screens_worker; a miss computes off the event
loop, bounded by the shared waveform semaphore. Returns 202 while audio
separation is still producing the stem (frontend polls, mirroring
/audio/vocals).
Response: {peaks_b64 (base64 uint8 per bucket), encoding: "u8",
peaks_per_second, duration_seconds}.
"""
from backend.services.audio_analysis_service import AudioAnalysisService
from backend.services.audio_transcoding_service import AudioTranscodingService
peaks_per_second = max(50, min(peaks_per_second, 1000))
job = await asyncio.to_thread(JobManager().get_job, job_id)
if not job:
raise HTTPException(status_code=404, detail=t("en", "review.jobNotFound"))
vocals_path = _vocals_stem_path(job)
if not vocals_path:
if not job.state_data.get("audio_complete", False):
return Response(status_code=202, headers={"Retry-After": "15"})
raise HTTPException(status_code=404, detail=t("en", "review.audioNotFound"))
try:
analysis_service = AudioAnalysisService()
async with _waveform_semaphore:
payload = await asyncio.to_thread(
analysis_service.get_vocals_peaks,
vocals_path,
job_id,
peaks_per_second,
AudioTranscodingService(),
)
return JSONResponse(
content={
"peaks_b64": payload["peaks_b64"],
"encoding": payload.get("encoding", "u8"),
"peaks_per_second": payload["peaks_per_second"],
"duration_seconds": payload["duration_seconds"],
},
# Immutable per stem path; short-lived like the audio responses.
headers={"Cache-Control": "private, max-age=600"},
)
except Exception as e:
logger.error(f"Job {job_id}: Error generating vocals peaks: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=t("en", "review.waveformGenerationError", error=str(e)))
@router.get("/{job_id}/waveform-data")
async def get_waveform_data(
job_id: str,
@@ -328,6 +328,86 @@ class AudioAnalysisService:
logger.warning(f"Failed to load review waveform cache {cache_gcs_path}: {e}")
return None
@staticmethod
def vocals_peaks_cache_path(job_id: str, peaks_per_second: int) -> str:
"""GCS path of the cached vocals peak-envelope JSON for a job."""
return f"jobs/{job_id}/review-audio/vocals_peaks_{peaks_per_second}.json"
def get_vocals_peaks(
self,
gcs_audio_path: str,
job_id: str,
peaks_per_second: int = 400,
transcoding_service=None,
) -> dict:
"""
Pre-computed peak envelope of the vocal stem for the review page's
Waveforms mode, cached persistently in GCS.
Returns the serve-ready payload::
{peaks_b64, encoding: "u8", peaks_per_second, duration_seconds,
source_gcs_path}
``peaks_b64`` is base64 of one uint8 per bucket (max-abs amplitude
quantized to 0-255) — ~150 KB for a 5-minute track vs the ~7 MB OGG
the frontend previously downloaded and decoded client-side. On a cache
miss, decode the small transcoded review OGG when it exists.
The cache upload is best-effort.
"""
import base64
cache_path = self.vocals_peaks_cache_path(job_id, peaks_per_second)
try:
if self.storage_service.file_exists(cache_path):
data = self.storage_service.download_json(cache_path)
if (
data.get("source_gcs_path") == gcs_audio_path
and data.get("peaks_per_second") == peaks_per_second
and data.get("peaks_b64")
):
logger.info(f"[{job_id}] Vocals peaks cache hit: {cache_path}")
return data
except Exception as e:
logger.warning(f"[{job_id}] Failed to load vocals peaks cache: {e}")
decode_path = gcs_audio_path
if transcoding_service is not None:
try:
ogg_path = transcoding_service.get_transcoded_cache_path(gcs_audio_path)
if self.storage_service.file_exists(ogg_path):
decode_path = ogg_path
except Exception as e:
logger.warning(f"[{job_id}] Could not resolve transcoded OGG for peaks: {e}")
import numpy as np
with tempfile.TemporaryDirectory() as temp_dir:
local_audio_path = os.path.join(temp_dir, local_audio_filename(decode_path, "vocals"))
self.storage_service.download_file(decode_path, local_audio_path)
peaks, duration = self.waveform_generator.generate_peaks(
local_audio_path, peaks_per_second=peaks_per_second
)
quantized = np.clip(np.round(peaks * 255.0), 0, 255).astype(np.uint8)
payload = {
"peaks_b64": base64.b64encode(quantized.tobytes()).decode("ascii"),
"encoding": "u8",
"peaks_per_second": peaks_per_second,
"duration_seconds": duration,
# Keyed to the LOGICAL source (the stem path) so a regenerated stem
# at a new path invalidates, while the OGG decode shortcut doesn't.
"source_gcs_path": gcs_audio_path,
}
try:
self.storage_service.upload_json(cache_path, payload)
logger.info(f"[{job_id}] Vocals peaks cached at {cache_path}")
except Exception as e:
logger.warning(f"[{job_id}] Failed to cache vocals peaks (non-fatal): {e}")
return payload
def cache_waveform_data(
self,
gcs_audio_path: str,
@@ -556,3 +556,105 @@ class TestGetReviewWaveform:
)
assert amps == [0.1, 0.9]
assert duration == 200.0
class TestGetVocalsPeaks:
"""Persistent GCS cache of the vocals peak envelope (sub-second strips)."""
def _service(self, mock_storage_class, mock_waveform_class):
import numpy as np
from backend.services.audio_analysis_service import AudioAnalysisService
mock_storage = Mock()
mock_storage_class.return_value = mock_storage
mock_waveform = Mock()
mock_waveform.generate_peaks.return_value = (
np.array([0.0, 0.5, 1.0], dtype=np.float32), 3.0
)
mock_waveform_class.return_value = mock_waveform
return AudioAnalysisService(), mock_storage, mock_waveform
@patch("backend.services.audio_analysis_service.StorageService")
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
@patch("backend.services.audio_analysis_service.WaveformGenerator")
def test_cache_hit_skips_decode(
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
):
service, storage, waveform = self._service(mock_storage_class, mock_waveform_class)
cached = {
"peaks_b64": "AAA=",
"encoding": "u8",
"peaks_per_second": 400,
"duration_seconds": 3.0,
"source_gcs_path": "jobs/j1/stems/vocals_clean.flac",
}
storage.file_exists.return_value = True
storage.download_json.return_value = cached
result = service.get_vocals_peaks("jobs/j1/stems/vocals_clean.flac", "j1", 400)
assert result == cached
waveform.generate_peaks.assert_not_called()
storage.upload_json.assert_not_called()
@patch("backend.services.audio_analysis_service.StorageService")
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
@patch("backend.services.audio_analysis_service.WaveformGenerator")
def test_cache_miss_computes_quantizes_and_uploads(
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
):
import base64
service, storage, waveform = self._service(mock_storage_class, mock_waveform_class)
storage.file_exists.return_value = False
result = service.get_vocals_peaks("jobs/j1/stems/vocals_clean.flac", "j1", 400)
assert result["encoding"] == "u8"
assert result["peaks_per_second"] == 400
assert result["duration_seconds"] == 3.0
# [0.0, 0.5, 1.0] -> uint8 [0, 128, 255]
assert list(base64.b64decode(result["peaks_b64"])) == [0, 128, 255]
cache_path, payload = storage.upload_json.call_args[0]
assert cache_path == "jobs/j1/review-audio/vocals_peaks_400.json"
assert payload == result
@patch("backend.services.audio_analysis_service.StorageService")
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
@patch("backend.services.audio_analysis_service.WaveformGenerator")
def test_stale_source_recomputes(
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
):
service, storage, waveform = self._service(mock_storage_class, mock_waveform_class)
storage.file_exists.return_value = True
storage.download_json.return_value = {
"peaks_b64": "AAA=",
"peaks_per_second": 400,
"duration_seconds": 3.0,
"source_gcs_path": "jobs/j1/stems/OLD.flac",
}
service.get_vocals_peaks("jobs/j1/stems/vocals_clean.flac", "j1", 400)
waveform.generate_peaks.assert_called_once()
@patch("backend.services.audio_analysis_service.StorageService")
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
@patch("backend.services.audio_analysis_service.WaveformGenerator")
def test_prefers_transcoded_ogg_and_survives_upload_failure(
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
):
service, storage, waveform = self._service(mock_storage_class, mock_waveform_class)
storage.file_exists.side_effect = lambda p: p.endswith(".ogg")
storage.upload_json.side_effect = Exception("GCS write denied")
transcoding = Mock()
transcoding.get_transcoded_cache_path.return_value = "jobs/j1/review-audio/vocals_clean.ogg"
result = service.get_vocals_peaks(
"jobs/j1/stems/vocals_clean.flac", "j1", 400, transcoding
)
# Decoded from the small OGG, and data still returned despite upload failure
downloaded_path = storage.download_file.call_args[0][0]
assert downloaded_path == "jobs/j1/review-audio/vocals_clean.ogg"
assert result["source_gcs_path"] == "jobs/j1/stems/vocals_clean.flac"
+72
View File
@@ -2139,3 +2139,75 @@ class TestLocalPreviewRenderBound:
assert await review_module._render_preview_locally(MagicMock(), {}, False) == "hash2"
asyncio.run(scenario())
class TestVocalsPeaksEndpoint:
"""/vocals-peaks serves the precomputed envelope (sub-second strips)."""
def _run(self, stems=None, audio_complete=True, pps=400):
import asyncio
from backend.api.routes.review import get_vocals_peaks
job = MagicMock()
job.input_media_gcs_path = "jobs/j/input.flac"
if stems is None:
stems = {"vocals_clean": "jobs/j/stems/vocals_clean.flac"}
job.file_urls = {"stems": stems}
job.state_data = {"audio_complete": audio_complete}
job_manager = MagicMock()
job_manager.get_job.return_value = job
analysis = MagicMock()
analysis.get_vocals_peaks.return_value = {
"peaks_b64": "AAECAw==",
"encoding": "u8",
"peaks_per_second": 400,
"duration_seconds": 12.5,
"source_gcs_path": "jobs/j/stems/vocals_clean.flac",
}
with patch("backend.api.routes.review.JobManager", return_value=job_manager), patch(
"backend.services.audio_analysis_service.AudioAnalysisService",
return_value=analysis,
), patch(
"backend.services.audio_transcoding_service.AudioTranscodingService",
return_value=MagicMock(),
):
resp = asyncio.run(
get_vocals_peaks("j", peaks_per_second=pps, auth_info=("admin", "full"))
)
return resp, analysis
def test_serves_payload_with_cache_headers(self):
import json as jsonlib
resp, analysis = self._run()
assert resp.status_code == 200
body = jsonlib.loads(resp.body)
assert body["peaks_b64"] == "AAECAw=="
assert body["encoding"] == "u8"
assert body["duration_seconds"] == 12.5
assert "max-age" in resp.headers.get("Cache-Control", "")
args = analysis.get_vocals_peaks.call_args[0]
assert args[0] == "jobs/j/stems/vocals_clean.flac"
assert args[1] == "j"
assert args[2] == 400
def test_clamps_peaks_per_second(self):
_, analysis = self._run(pps=999999)
assert analysis.get_vocals_peaks.call_args[0][2] == 1000
def test_202_while_separation_running(self):
resp, analysis = self._run(stems={}, audio_complete=False)
assert resp.status_code == 202
assert resp.headers.get("Retry-After") == "15"
analysis.get_vocals_peaks.assert_not_called()
def test_404_when_stem_will_never_exist(self):
from fastapi import HTTPException
with pytest.raises(HTTPException) as exc_info:
self._run(stems={}, audio_complete=True)
assert exc_info.value.status_code == 404
+20
View File
@@ -0,0 +1,20 @@
"""Stem-selection helpers shared by review routes and workers."""
from typing import Optional
def vocals_stem_path(job) -> Optional[str]:
"""The full-vocal-mix stem for the review waveform, or None if absent.
Stem keys vary by separation model. We want the full vocal mix
(lead + backing) so every word can be lined up against the waveform:
- "vocals" : full vocal from a 2-stem split (rare)
- "vocals_clean" : full vocal from the primary vocal/instrumental
split (present on essentially all cloud jobs)
- "lead_vocals" : last-resort fallback (misses backing lines)
"""
stems = job.file_urls.get("stems", {}) if job.file_urls else {}
for key in ("vocals", "vocals_clean", "lead_vocals"):
if stems.get(key):
return stems[key]
return None
+17
View File
@@ -786,6 +786,23 @@ async def _transcode_review_audio(
# Non-fatal: /waveform-data computes + caches lazily on first open
job_log.warning(f"Review waveform pre-compute failed (non-fatal): {e}")
# Pre-compute the vocals peak envelope too (Waveforms review mode) so
# /vocals-peaks is a pure cache read and the per-segment strips paint
# sub-second on first open.
from backend.utils.stems import vocals_stem_path
vocals_source = vocals_stem_path(job)
if vocals_source:
try:
await asyncio.to_thread(
AudioAnalysisService().get_vocals_peaks,
vocals_source, job_id, 400, transcoding,
)
job_log.info("Vocals peak envelope pre-computed")
except Exception as e:
# Non-fatal: /vocals-peaks computes + caches lazily on first open
job_log.warning(f"Vocals peaks pre-compute failed (non-fatal): {e}")
except Exception as e:
# Non-fatal: review UI will fall back to FLAC signed URLs
job_log.warning(f"Review audio transcoding failed (non-fatal): {e}")
+17
View File
@@ -2696,3 +2696,20 @@ localhost no-op). Added in v0.231.0 — see
`docs/archive/2026-09-18-concurrent-review-reliability-investigation-and-plan.md`.
Crash reports are the separate `POST /api/client-errors` (error-monitor pipeline).
### Vocals Peaks (Waveforms review mode)
```http
GET /api/review/{job_id}/vocals-peaks?peaks_per_second=400
```
Pre-computed max-abs peak envelope of the vocal stem:
`{peaks_b64, encoding: "u8", peaks_per_second, duration_seconds}` — base64 of one
uint8 per 1/400s bucket (~150 KB for a 5-minute track). Replaces the frontend's
download+decode of the whole vocals OGG so Waveforms-mode strips paint sub-second.
Served from a persistent GCS cache (`review-audio/vocals_peaks_{pps}.json`,
validated against the stem path), pre-computed by `screens_worker`; a miss computes
off the event loop under the shared waveform semaphore. Returns `202 Retry-After: 15`
while audio separation is still producing the stem. Auth like `/audio/vocals`
(`?token=` / `?review_token=`). The frontend falls back to the full audio decode if
this endpoint fails (`VocalsAudioDataLoader`). Added in v0.232.0.
@@ -88,6 +88,9 @@ interface ApiClient {
searchLyrics?: (artist: string, title: string, forceSources?: string[]) => Promise<SearchLyricsResponse>
getAudioUrl: (hash: string) => string
getVocalsAudioUrl: () => string
// Optional: pre-computed peaks endpoint (cloud mode); absent → loader falls
// back to downloading + decoding the full stem.
getVocalsPeaksUrl?: () => string
generatePreviewVideo: (data: CorrectionData, isDuet?: boolean) => Promise<{
status: string
message?: string
@@ -177,6 +180,7 @@ export default function LyricsAnalyzer({
const [instrumentalChoice, setInstrumentalChoice] = useState<'clean' | 'with_backing' | null>(null)
const [vocalsAudioUrl, setVocalsAudioUrl] = useState<string | null>(null)
const [vocalsPeaksUrl, setVocalsPeaksUrl] = useState<string | null>(null)
// Success screen state (for auto-close after submission)
const [showSuccess, setShowSuccess] = useState(false)
@@ -657,6 +661,7 @@ export default function LyricsAnalyzer({
if (!apiClient) return
setVocalsAudioUrl(apiClient.getVocalsAudioUrl())
setVocalsPeaksUrl(apiClient.getVocalsPeaksUrl ? apiClient.getVocalsPeaksUrl() : null)
}, [apiClient])
// Countdown effect for success screen (auto-close/redirect after submission)
@@ -1532,7 +1537,7 @@ export default function LyricsAnalyzer({
}
return (
<VocalsAudioDataLoader audioUrl={vocalsAudioUrl}>
<VocalsAudioDataLoader audioUrl={vocalsAudioUrl} peaksUrl={vocalsPeaksUrl}>
<div className="max-w-full overflow-x-hidden">
<Header
isReadOnly={isReadOnly}
@@ -1,4 +1,4 @@
import { AudioData, AudioNotReadyError, fetchAudioData, isTransientAudioError } from '@/lib/audio-data'
import { AudioData, AudioNotReadyError, fetchAudioData, fetchVocalsPeaks, isTransientAudioError } from '@/lib/audio-data'
import { reportDegradationEvent } from '@/lib/degradation-events'
import { createContext, PropsWithChildren, useEffect, useState } from 'react'
@@ -11,6 +11,10 @@ export const VocalsAudioDataLoaderContext = createContext<{ audioData: AudioData
export interface AudioDataLoaderProps extends PropsWithChildren {
audioUrl: string | null
/** Pre-computed peak-envelope endpoint (~150 KB JSON). Preferred over
* downloading + decoding the whole stem from audioUrl; when it fails
* terminally the loader falls back to the full audio decode. */
peaksUrl?: string | null
}
// Audio separation runs in the background while the user reviews lyrics, so the
@@ -25,11 +29,11 @@ const NOT_READY_MAX_RETRIES = 40 // 40 × 15s = 10 minutes
// Waveforms view stripless until a manual reload. Retry a few times with backoff.
const TRANSIENT_RETRY_DELAYS_MS = [5_000, 15_000, 30_000, 60_000]
export const VocalsAudioDataLoader = ({ audioUrl, children }: AudioDataLoaderProps) => {
export const VocalsAudioDataLoader = ({ audioUrl, peaksUrl, children }: AudioDataLoaderProps) => {
const [audioData, setAudioData] = useState<AudioData | null>(null)
useEffect(() => {
if (!audioUrl) return
if (!audioUrl && !peaksUrl) return
// Guard against (a) an unhandled rejection when the endpoint 404s (no vocal
// stem for this job) and (b) a stale in-flight fetch resolving after a newer
@@ -38,8 +42,10 @@ export const VocalsAudioDataLoader = ({ audioUrl, children }: AudioDataLoaderPro
let retryTimer: ReturnType<typeof setTimeout> | undefined
const startedAt = Date.now()
const load = (attempt: number, transientAttempt: number) => {
fetchAudioData(audioUrl)
const load = (attempt: number, transientAttempt: number, usePeaks: boolean) => {
const source =
usePeaks && peaksUrl ? fetchVocalsPeaks(peaksUrl) : fetchAudioData(audioUrl!)
source
.then((audioData) => {
if (cancelled) return
const elapsedMs = Date.now() - startedAt
@@ -48,6 +54,7 @@ export const VocalsAudioDataLoader = ({ audioUrl, children }: AudioDataLoaderPro
elapsed_ms: elapsedMs,
not_ready_polls: attempt,
transient_retries: transientAttempt,
used_peaks: usePeaks,
})
}
setAudioData(audioData)
@@ -55,13 +62,25 @@ export const VocalsAudioDataLoader = ({ audioUrl, children }: AudioDataLoaderPro
.catch((error) => {
if (cancelled) return
if (error instanceof AudioNotReadyError && attempt < NOT_READY_MAX_RETRIES) {
retryTimer = setTimeout(() => load(attempt + 1, transientAttempt), NOT_READY_RETRY_MS)
retryTimer = setTimeout(() => load(attempt + 1, transientAttempt, usePeaks), NOT_READY_RETRY_MS)
return
}
if (isTransientAudioError(error) && transientAttempt < TRANSIENT_RETRY_DELAYS_MS.length) {
// Peaks get ONE quick retry, then we fall back to the audio path
// (which has the full backoff ladder) — a flaky peaks endpoint
// must not delay strips by the whole retry ladder first.
const maxTransientRetries = usePeaks ? 1 : TRANSIENT_RETRY_DELAYS_MS.length
if (isTransientAudioError(error) && transientAttempt < maxTransientRetries) {
const delay = TRANSIENT_RETRY_DELAYS_MS[transientAttempt]
console.warn(`Vocals audio load failed, retrying in ${delay / 1000}s`, error)
retryTimer = setTimeout(() => load(attempt, transientAttempt + 1), delay)
console.warn(`Vocals ${usePeaks ? 'peaks' : 'audio'} load failed, retrying in ${delay / 1000}s`, error)
retryTimer = setTimeout(() => load(attempt, transientAttempt + 1, usePeaks), delay)
return
}
if (usePeaks && audioUrl) {
// Peaks endpoint terminally failed (or exhausted retries) —
// fall back to the original full-download-and-decode path so a
// peaks-side problem never costs the user their waveforms.
console.warn('Vocals peaks unavailable, falling back to full audio decode', error)
load(attempt, 0, false)
return
}
console.error('Failed to load vocals audio data', error)
@@ -75,14 +94,14 @@ export const VocalsAudioDataLoader = ({ audioUrl, children }: AudioDataLoaderPro
})
}
load(0, 0)
load(0, 0, Boolean(peaksUrl))
return () => {
cancelled = true
if (retryTimer) clearTimeout(retryTimer)
setAudioData(null)
}
}, [audioUrl])
}, [audioUrl, peaksUrl])
return (
<VocalsAudioDataLoaderContext.Provider value={{ audioData }}>
@@ -38,6 +38,9 @@ function makeMockApiClient(): LyricsReviewApiClient {
searchLyrics: jest.fn(),
getAudioUrl: jest.fn(),
getVocalsAudioUrl: jest.fn(),
getVocalsPeaksUrl: jest.fn(),
getWaveformData: jest.fn(),
refreshInstrumentalUrls: jest.fn(),
generatePreviewVideo: jest.fn(),
getPreviewVideoStatus: jest.fn(),
getPreviewVideoUrl: jest.fn(),
@@ -5,7 +5,7 @@ import { VocalsAudioDataLoader, VocalsAudioDataLoaderContext } from '../VocalsAu
jest.mock('@/lib/audio-data', () => {
const actual = jest.requireActual('@/lib/audio-data')
return { ...actual, fetchAudioData: jest.fn() }
return { ...actual, fetchAudioData: jest.fn(), fetchVocalsPeaks: jest.fn() }
})
const fetchAudioData = fetchAudioDataImport as jest.Mock
@@ -185,3 +185,108 @@ describe('VocalsAudioDataLoader', () => {
expect(fetchAudioData).toHaveBeenCalledTimes(1)
})
})
describe('VocalsAudioDataLoader peaks-first behavior', () => {
const audioDataLib = jest.requireMock('@/lib/audio-data')
beforeEach(() => {
jest.clearAllMocks()
jest.useFakeTimers()
audioDataLib.fetchVocalsPeaks = jest.fn()
})
afterEach(() => {
jest.useRealTimers()
})
it('prefers the peaks endpoint and never downloads the full audio', async () => {
audioDataLib.fetchVocalsPeaks.mockResolvedValue(AUDIO_DATA)
render(
<VocalsAudioDataLoader audioUrl="https://api/audio/vocals" peaksUrl="https://api/vocals-peaks">
<Probe />
</VocalsAudioDataLoader>
)
await flushPromises()
expect(screen.getByTestId('probe')).toHaveTextContent('loaded')
expect(audioDataLib.fetchVocalsPeaks).toHaveBeenCalledTimes(1)
expect(fetchAudioData).not.toHaveBeenCalled()
})
it('falls back to full audio decode when peaks fail terminally', async () => {
audioDataLib.fetchVocalsPeaks.mockRejectedValue(new AudioFetchError(404))
fetchAudioData.mockResolvedValue(AUDIO_DATA)
const consoleWarn = jest.spyOn(console, 'warn').mockImplementation(() => {})
render(
<VocalsAudioDataLoader audioUrl="https://api/audio/vocals" peaksUrl="https://api/vocals-peaks">
<Probe />
</VocalsAudioDataLoader>
)
await flushPromises()
expect(fetchAudioData).toHaveBeenCalledTimes(1)
expect(screen.getByTestId('probe')).toHaveTextContent('loaded')
consoleWarn.mockRestore()
})
it('retries peaks once on a transient error, then falls back to audio', async () => {
audioDataLib.fetchVocalsPeaks.mockRejectedValue(new AudioFetchError(503))
fetchAudioData.mockResolvedValue(AUDIO_DATA)
const consoleWarn = jest.spyOn(console, 'warn').mockImplementation(() => {})
render(
<VocalsAudioDataLoader audioUrl="https://api/audio/vocals" peaksUrl="https://api/vocals-peaks">
<Probe />
</VocalsAudioDataLoader>
)
await flushPromises()
expect(audioDataLib.fetchVocalsPeaks).toHaveBeenCalledTimes(1)
// One quick retry of peaks...
await act(async () => {
jest.advanceTimersByTime(5_000)
})
expect(audioDataLib.fetchVocalsPeaks).toHaveBeenCalledTimes(2)
// ...then immediate fallback to the audio path (no more peaks calls).
expect(fetchAudioData).toHaveBeenCalledTimes(1)
expect(screen.getByTestId('probe')).toHaveTextContent('loaded')
consoleWarn.mockRestore()
})
it('polls peaks on 202 while separation is running', async () => {
audioDataLib.fetchVocalsPeaks
.mockRejectedValueOnce(new AudioNotReadyError())
.mockResolvedValue(AUDIO_DATA)
render(
<VocalsAudioDataLoader audioUrl="https://api/audio/vocals" peaksUrl="https://api/vocals-peaks">
<Probe />
</VocalsAudioDataLoader>
)
await flushPromises()
expect(screen.getByTestId('probe')).toHaveTextContent('empty')
await act(async () => {
jest.advanceTimersByTime(15_000)
})
expect(screen.getByTestId('probe')).toHaveTextContent('loaded')
expect(fetchAudioData).not.toHaveBeenCalled()
})
it('works with only audioUrl (local mode without a peaks endpoint)', async () => {
fetchAudioData.mockResolvedValue(AUDIO_DATA)
render(
<VocalsAudioDataLoader audioUrl="https://api/audio/vocals">
<Probe />
</VocalsAudioDataLoader>
)
await flushPromises()
expect(screen.getByTestId('probe')).toHaveTextContent('loaded')
expect(audioDataLib.fetchVocalsPeaks).not.toHaveBeenCalled()
})
})
@@ -127,4 +127,32 @@ test.describe('Review fast-full-load contract', () => {
expect(data.amplitudes.length).toBeGreaterThan(0);
expect(data.duration_seconds).toBeGreaterThan(0);
});
// Sub-second strips (v0.232.0): precomputed vocals peak envelope.
test('vocals-peaks returns a compact cached envelope quickly', async ({ request }) => {
const base = `${API_URL}/api/review/${JOB_ID}/vocals-peaks`;
const url = REVIEW_TOKEN ? `${base}?review_token=${encodeURIComponent(REVIEW_TOKEN)}` : base;
// First call may compute+cache; the second must be a pure cache read.
const warm = await request.get(url, { headers: authHeaders() });
expect([200, 202]).toContain(warm.status());
if (warm.status() === 202) return; // separation still running for this job
const started = Date.now();
const res = await request.get(url, { headers: authHeaders() });
const elapsedMs = Date.now() - started;
console.log(`vocals-peaks (cached) status=${res.status()} elapsed=${elapsedMs}ms`);
expect(res.ok()).toBeTruthy();
expect(elapsedMs).toBeLessThan(5_000);
expect(res.headers()['cache-control']).toContain('max-age');
const data = await res.json();
expect(data.encoding).toBe('u8');
expect(typeof data.peaks_b64).toBe('string');
expect(data.peaks_b64.length).toBeGreaterThan(0);
expect(data.peaks_per_second).toBe(400);
expect(data.duration_seconds).toBeGreaterThan(0);
// Envelope should be dramatically smaller than the audio it replaces.
expect(data.peaks_b64.length).toBeLessThan(1_000_000);
});
});
+44
View File
@@ -116,3 +116,47 @@ describe('computePeaks', () => {
expect(peaks[0]).toBeCloseTo(0.5)
})
})
describe('fetchVocalsPeaks', () => {
const { fetchVocalsPeaks, AudioNotReadyError, AudioFetchError } = jest.requireActual('@/lib/audio-data')
const mockJsonResponse = (status: number, body?: unknown) =>
({ ok: status >= 200 && status < 300, status, json: async () => body }) as Response
afterEach(() => {
jest.restoreAllMocks()
})
it('decodes the u8/base64 payload into AudioData', async () => {
// peaks [0, 128, 255] -> [0, ~0.5, 1]
const b64 = btoa(String.fromCharCode(0, 128, 255))
global.fetch = jest.fn(async () =>
mockJsonResponse(200, {
peaks_b64: b64,
encoding: 'u8',
peaks_per_second: 400,
duration_seconds: 0.0075,
})
) as unknown as typeof fetch
const data = await fetchVocalsPeaks('https://api/vocals-peaks')
expect(data.peaksPerSecond).toBe(400)
expect(data.duration).toBeCloseTo(0.0075)
expect(Array.from(data.peaks)).toEqual([0, Math.fround(128 / 255), 1])
})
it('throws AudioNotReadyError on 202', async () => {
global.fetch = jest.fn(async () => mockJsonResponse(202)) as unknown as typeof fetch
await expect(fetchVocalsPeaks('u')).rejects.toBeInstanceOf(AudioNotReadyError)
})
it('throws AudioFetchError with status on non-2xx', async () => {
global.fetch = jest.fn(async () => mockJsonResponse(404)) as unknown as typeof fetch
await expect(fetchVocalsPeaks('u')).rejects.toMatchObject({ status: 404 })
})
it('treats a malformed payload as a transient server error', async () => {
global.fetch = jest.fn(async () => mockJsonResponse(200, { nope: true })) as unknown as typeof fetch
await expect(fetchVocalsPeaks('u')).rejects.toBeInstanceOf(AudioFetchError)
})
})
+11
View File
@@ -3921,6 +3921,7 @@ export interface LyricsReviewApiClient {
searchLyrics: (artist: string, title: string, forceSources?: string[]) => Promise<SearchLyricsResponse>
getAudioUrl: (hash: string) => string
getVocalsAudioUrl: () => string
getVocalsPeaksUrl: () => string
generatePreviewVideo: (data: CorrectionData, isDuet?: boolean) => Promise<{
status: string
message?: string
@@ -4075,6 +4076,16 @@ export function createLyricsReviewApiClient(jobId: string): LyricsReviewApiClien
return token ? `${base}?token=${encodeURIComponent(token)}` : base
},
/**
* Pre-computed vocals peak-envelope URL (Waveforms review mode).
* ~150 KB JSON instead of downloading + decoding the whole vocal stem.
*/
getVocalsPeaksUrl(): string {
const token = getAccessToken()
const base = `${API_BASE_URL}/api/review/${jobId}/vocals-peaks`
return token ? `${base}?token=${encodeURIComponent(token)}` : base
},
/**
* Generate a preview video from the current correction data.
*
+31
View File
@@ -93,6 +93,37 @@ export function computePeaks(decoded: DecodedAudioLike): AudioData {
}
}
/**
* Fetch the server-side pre-computed vocals peak envelope (~150 KB JSON) and
* adapt it to the same AudioData shape computePeaks produces — so waveform
* strips paint sub-second instead of after a multi-MB download + decode.
* Throws AudioNotReadyError on 202 (separation still running) and
* AudioFetchError on other non-2xx, mirroring fetchAudioData.
*/
export async function fetchVocalsPeaks(url: string): Promise<AudioData> {
const response = await fetch(url)
if (response.status === 202) {
throw new AudioNotReadyError()
}
if (!response.ok) {
throw new AudioFetchError(response.status)
}
const data = await response.json()
if (data?.encoding !== 'u8' || typeof data.peaks_b64 !== 'string' || !data.peaks_b64) {
throw new AudioFetchError(500) // unexpected payload — treat as transient server trouble
}
const raw = atob(data.peaks_b64)
const peaks = new Float32Array(raw.length)
for (let i = 0; i < raw.length; i++) {
peaks[i] = raw.charCodeAt(i) / 255
}
return {
duration: Number(data.duration_seconds) || 0,
peaks,
peaksPerSecond: Number(data.peaks_per_second) || PEAKS_PER_SECOND,
}
}
export async function fetchAudioData(url: string): Promise<AudioData> {
// fetch() resolves even for 4xx/5xx; the vocals endpoint 404s when a job has
// no vocal stem. Fail explicitly so the caller sees "no vocals" rather than an
+54 -1
View File
@@ -208,7 +208,60 @@ class WaveformGenerator:
amplitudes.append(normalized)
return amplitudes, duration_seconds
def generate_peaks(
self,
audio_path: str,
peaks_per_second: int = 400,
) -> Tuple[np.ndarray, float]:
"""
Generate a max-abs peak envelope at a fixed time resolution.
Mirrors the review frontend's ``computePeaks`` (lib/audio-data.ts):
one peak per 1/peaks_per_second bucket, each the maximum absolute
sample value across all channels, normalized to 0.0–1.0. Serving this
pre-computed lets the review page draw per-segment waveform strips
without downloading + decoding the whole vocal stem client-side.
Vectorized (numpy) — a 4-minute stereo track computes in well under a
second, vs the pure-Python RMS loop in generate_data_only.
Returns:
Tuple of (peaks float32 ndarray in 0..1, duration_seconds)
"""
path = Path(audio_path)
if not path.exists():
raise FileNotFoundError(f"Audio file not found: {audio_path}")
audio = AudioSegment.from_file(audio_path)
duration_seconds = len(audio) / 1000.0
samples = np.abs(
np.asarray(audio.get_array_of_samples(), dtype=np.float32)
) / float(audio.max_possible_amplitude)
if audio.channels > 1:
# Interleaved -> (frames, channels) -> per-frame max across channels
frames = len(samples) // audio.channels
samples = samples[: frames * audio.channels]
samples = samples.reshape(frames, audio.channels).max(axis=1)
bucket_count = max(1, math.ceil(duration_seconds * peaks_per_second))
# Fractional stride with floored boundaries, matching the frontend's
# drift-free bucketing (see the 44.1kHz drift fix notes there).
samples_per_bucket = len(samples) / bucket_count
boundaries = np.floor(np.arange(bucket_count + 1) * samples_per_bucket).astype(np.int64)
boundaries[-1] = len(samples)
peaks = np.zeros(bucket_count, dtype=np.float32)
# maximum.reduceat needs strictly valid starts; guard empty buckets.
starts = boundaries[:-1]
if len(samples):
reduced = np.maximum.reduceat(samples, np.minimum(starts, len(samples) - 1))
widths = boundaries[1:] - starts
peaks = np.where(widths > 0, reduced, 0.0).astype(np.float32)
return peaks, duration_seconds
def _get_envelope(
self,
audio: AudioSegment,
+1 -1
View File
@@ -1,6 +1,6 @@
[tool.poetry]
name = "karaoke-gen"
version = "0.231.1"
version = "0.232.0"
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
license = "MIT"
@@ -0,0 +1,59 @@
"""Tests for WaveformGenerator.generate_peaks (review Waveforms-mode envelope)."""
import math
import tempfile
import numpy as np
import pytest
from pydub import AudioSegment
from pydub.generators import Sine
from karaoke_gen.instrumental_review.waveform import WaveformGenerator
@pytest.fixture(scope="module")
def stereo_wav(tmp_path_factory):
"""3s stereo: -6dB 440Hz left, -20dB 880Hz right."""
left = Sine(440).to_audio_segment(duration=3000).apply_gain(-6)
right = Sine(880).to_audio_segment(duration=3000).apply_gain(-20)
seg = AudioSegment.from_mono_audiosegments(left, right)
path = tmp_path_factory.mktemp("audio") / "stereo.wav"
seg.export(str(path), format="wav")
return str(path)
class TestGeneratePeaks:
def test_bucket_count_and_duration(self, stereo_wav):
peaks, duration = WaveformGenerator().generate_peaks(stereo_wav, peaks_per_second=400)
assert duration == pytest.approx(3.0, abs=0.05)
assert len(peaks) == math.ceil(duration * 400)
def test_peaks_are_max_abs_across_channels_normalized(self, stereo_wav):
peaks, _ = WaveformGenerator().generate_peaks(stereo_wav, peaks_per_second=400)
# The louder channel (-6 dB sine) dominates: peak amplitude ~10^(-6/20)=0.501
assert float(peaks.max()) == pytest.approx(0.501, abs=0.01)
assert float(peaks.min()) >= 0.0
assert peaks.dtype == np.float32
def test_matches_naive_reference(self, stereo_wav):
pps = 400
peaks, duration = WaveformGenerator().generate_peaks(stereo_wav, peaks_per_second=pps)
audio = AudioSegment.from_file(stereo_wav)
s = np.abs(np.asarray(audio.get_array_of_samples(), dtype=np.float32)) / audio.max_possible_amplitude
frames = len(s) // audio.channels
s = s[: frames * audio.channels].reshape(frames, audio.channels).max(axis=1)
bucket_count = math.ceil(duration * pps)
spb = len(s) / bucket_count
ref = np.array(
[
s[math.floor(i * spb): (len(s) if i == bucket_count - 1 else max(math.floor((i + 1) * spb), math.floor(i * spb) + 1))].max()
for i in range(bucket_count)
],
dtype=np.float32,
)
assert np.abs(ref - peaks).max() == 0.0
def test_missing_file_raises(self):
with pytest.raises(FileNotFoundError):
WaveformGenerator().generate_peaks("/nonexistent.wav")