mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 13:07:57 +09:00
feat: precomputed vocals peaks — Waveforms strips paint sub-second (#1031)
## Summary Final piece of the review-page speed work (#1024 → #1028 → #1029): the Waveforms-mode strips no longer require downloading + WebAudio-decoding the entire vocals stem (~7.5MB, ~4–6s) — the backend now serves a **precomputed 400-peaks/s envelope as ~150KB of uint8/base64 JSON**, cached in GCS and pre-warmed at prep time, so strips paint sub-second. ## How - **`WaveformGenerator.generate_peaks()`** — numpy max-abs envelope, bucket-for-bucket identical to the frontend's `computePeaks` (same drift-free fractional-stride bucketing; exact match vs a naive reference in tests). - **`AudioAnalysisService.get_vocals_peaks()`** — GCS cache at `review-audio/vocals_peaks_400.json`, validated against the stem path; miss decodes the small transcoded OGG; upload best-effort. - **`GET /api/review/{job_id}/vocals-peaks`** — off the event loop under the shared waveform semaphore; `202 Retry-After: 15` while separation runs (mirrors `/audio/vocals`); `Cache-Control: private, max-age=600`; `peaks_per_second` clamped 50–1000. - **`screens_worker`** pre-computes the envelope next to the existing OGG + waveform pre-warm, so it's a pure cache read by the time a job reaches review. - **Frontend** — `VocalsAudioDataLoader` prefers the peaks endpoint (one quick retry on transient errors) and **falls back to the original full download+decode** on any terminal peaks failure, so a peaks-side problem can never cost a user their waveforms. Local mode (no peaks URL) uses the old path unchanged. Vocal stem selection factored into `backend/utils/stems.py` (shared by routes + worker). ## Testing - Package: 4 tests incl. exact naive-reference parity (**106 passed**) - Backend: 4 service + 4 endpoint tests (**4518 passed**) - Frontend: `fetchVocalsPeaks` units + 5 loader-behavior tests (prefers peaks / falls back / one-retry-then-fallback / 202 polling / audio-only mode) (**1307 passed**) - `review-fast-load.spec.ts` prod E2E extended with a cached vocals-peaks latency/shape check - Post-deploy: will verify the endpoint on a real in-review job and confirm payload size + latency ## Notes - Existing in-review jobs have no cache yet — their first Waveforms open computes + caches lazily (bounded); every open after that is sub-second. New jobs are pre-warmed. - 8-bit amplitude quantization is visually lossless at strip heights (14px). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
7cb5de729a
commit
d85b6e271c
18 files changed
+744
-28
No files matched your search
@@ -25,7 +25,7 @@ from pathlib import Path
|
||||
from typing import Dict, Any, List, Literal, Optional, Set, Tuple
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Request, Depends, Form, File, UploadFile
|
||||
from fastapi.responses import RedirectResponse, Response
|
||||
from fastapi.responses import JSONResponse, RedirectResponse, Response
|
||||
from google.cloud.exceptions import NotFound
|
||||
from google.cloud.firestore_v1.field_path import FieldPath
|
||||
|
||||
@@ -38,6 +38,7 @@ from backend.services.job_logging import job_log_context, JobLogger
|
||||
from backend.services.tracing import create_span, add_span_attribute, add_span_event
|
||||
from backend.services.encoding_service import get_encoding_service
|
||||
from backend.api.dependencies import require_auth, require_review_auth
|
||||
from backend.utils.stems import vocals_stem_path as _vocals_stem_path
|
||||
from backend.services.auth_service import UserType
|
||||
from backend.config import get_settings
|
||||
from backend.i18n import t, get_locale_from_request
|
||||
@@ -874,18 +875,7 @@ async def _stream_audio(job_id: str, request: Optional[Request] = None, stem: Li
|
||||
if stem == "input":
|
||||
audio_gcs_path = job.input_media_gcs_path
|
||||
elif stem == "vocals":
|
||||
stems = job.file_urls.get("stems", {}) if job.file_urls else {}
|
||||
# Stem keys vary by separation model. We want the full vocal mix
|
||||
# (lead + backing) so every word can be lined up against the waveform:
|
||||
# - "vocals" : full vocal from a 2-stem split (rare)
|
||||
# - "vocals_clean" : full vocal from the primary vocal/instrumental
|
||||
# split (present on essentially all cloud jobs)
|
||||
# - "lead_vocals" : last-resort fallback (misses backing lines)
|
||||
# Use .get() so a missing key yields a clean 404 below, not a KeyError/500.
|
||||
for key in ("vocals", "vocals_clean", "lead_vocals"):
|
||||
if stems.get(key):
|
||||
audio_gcs_path = stems[key]
|
||||
break
|
||||
audio_gcs_path = _vocals_stem_path(job)
|
||||
if not audio_gcs_path:
|
||||
# Audio separation is decoupled from the lyrics-review critical path, so a
|
||||
# user can open review before the vocal stems have been uploaded (the stem
|
||||
@@ -2201,6 +2191,66 @@ _WAVEFORM_MAX_CONCURRENCY = max(1, int(os.getenv("REVIEW_WAVEFORM_MAX_CONCURRENC
|
||||
_waveform_semaphore = asyncio.Semaphore(_WAVEFORM_MAX_CONCURRENCY)
|
||||
|
||||
|
||||
@router.get("/{job_id}/vocals-peaks")
|
||||
async def get_vocals_peaks(
|
||||
job_id: str,
|
||||
peaks_per_second: int = 400,
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""
|
||||
Pre-computed peak envelope of the vocal stem (Waveforms review mode).
|
||||
|
||||
Replaces the frontend's download-and-decode of the whole vocals OGG
|
||||
(~7 MB + WebAudio decode) with a ~150 KB JSON, so the per-segment
|
||||
waveform strips paint sub-second on page load. Served from a persistent
|
||||
GCS cache, pre-computed by screens_worker; a miss computes off the event
|
||||
loop, bounded by the shared waveform semaphore. Returns 202 while audio
|
||||
separation is still producing the stem (frontend polls, mirroring
|
||||
/audio/vocals).
|
||||
|
||||
Response: {peaks_b64 (base64 uint8 per bucket), encoding: "u8",
|
||||
peaks_per_second, duration_seconds}.
|
||||
"""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
from backend.services.audio_transcoding_service import AudioTranscodingService
|
||||
|
||||
peaks_per_second = max(50, min(peaks_per_second, 1000))
|
||||
|
||||
job = await asyncio.to_thread(JobManager().get_job, job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=t("en", "review.jobNotFound"))
|
||||
|
||||
vocals_path = _vocals_stem_path(job)
|
||||
if not vocals_path:
|
||||
if not job.state_data.get("audio_complete", False):
|
||||
return Response(status_code=202, headers={"Retry-After": "15"})
|
||||
raise HTTPException(status_code=404, detail=t("en", "review.audioNotFound"))
|
||||
|
||||
try:
|
||||
analysis_service = AudioAnalysisService()
|
||||
async with _waveform_semaphore:
|
||||
payload = await asyncio.to_thread(
|
||||
analysis_service.get_vocals_peaks,
|
||||
vocals_path,
|
||||
job_id,
|
||||
peaks_per_second,
|
||||
AudioTranscodingService(),
|
||||
)
|
||||
return JSONResponse(
|
||||
content={
|
||||
"peaks_b64": payload["peaks_b64"],
|
||||
"encoding": payload.get("encoding", "u8"),
|
||||
"peaks_per_second": payload["peaks_per_second"],
|
||||
"duration_seconds": payload["duration_seconds"],
|
||||
},
|
||||
# Immutable per stem path; short-lived like the audio responses.
|
||||
headers={"Cache-Control": "private, max-age=600"},
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Error generating vocals peaks: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=t("en", "review.waveformGenerationError", error=str(e)))
|
||||
|
||||
|
||||
@router.get("/{job_id}/waveform-data")
|
||||
async def get_waveform_data(
|
||||
job_id: str,
|
||||
|
||||
@@ -328,6 +328,86 @@ class AudioAnalysisService:
|
||||
logger.warning(f"Failed to load review waveform cache {cache_gcs_path}: {e}")
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def vocals_peaks_cache_path(job_id: str, peaks_per_second: int) -> str:
|
||||
"""GCS path of the cached vocals peak-envelope JSON for a job."""
|
||||
return f"jobs/{job_id}/review-audio/vocals_peaks_{peaks_per_second}.json"
|
||||
|
||||
def get_vocals_peaks(
|
||||
self,
|
||||
gcs_audio_path: str,
|
||||
job_id: str,
|
||||
peaks_per_second: int = 400,
|
||||
transcoding_service=None,
|
||||
) -> dict:
|
||||
"""
|
||||
Pre-computed peak envelope of the vocal stem for the review page's
|
||||
Waveforms mode, cached persistently in GCS.
|
||||
|
||||
Returns the serve-ready payload::
|
||||
|
||||
{peaks_b64, encoding: "u8", peaks_per_second, duration_seconds,
|
||||
source_gcs_path}
|
||||
|
||||
``peaks_b64`` is base64 of one uint8 per bucket (max-abs amplitude
|
||||
quantized to 0-255) — ~150 KB for a 5-minute track vs the ~7 MB OGG
|
||||
the frontend previously downloaded and decoded client-side. On a cache
|
||||
miss, decode the small transcoded review OGG when it exists.
|
||||
The cache upload is best-effort.
|
||||
"""
|
||||
import base64
|
||||
|
||||
cache_path = self.vocals_peaks_cache_path(job_id, peaks_per_second)
|
||||
try:
|
||||
if self.storage_service.file_exists(cache_path):
|
||||
data = self.storage_service.download_json(cache_path)
|
||||
if (
|
||||
data.get("source_gcs_path") == gcs_audio_path
|
||||
and data.get("peaks_per_second") == peaks_per_second
|
||||
and data.get("peaks_b64")
|
||||
):
|
||||
logger.info(f"[{job_id}] Vocals peaks cache hit: {cache_path}")
|
||||
return data
|
||||
except Exception as e:
|
||||
logger.warning(f"[{job_id}] Failed to load vocals peaks cache: {e}")
|
||||
|
||||
decode_path = gcs_audio_path
|
||||
if transcoding_service is not None:
|
||||
try:
|
||||
ogg_path = transcoding_service.get_transcoded_cache_path(gcs_audio_path)
|
||||
if self.storage_service.file_exists(ogg_path):
|
||||
decode_path = ogg_path
|
||||
except Exception as e:
|
||||
logger.warning(f"[{job_id}] Could not resolve transcoded OGG for peaks: {e}")
|
||||
|
||||
import numpy as np
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
local_audio_path = os.path.join(temp_dir, local_audio_filename(decode_path, "vocals"))
|
||||
self.storage_service.download_file(decode_path, local_audio_path)
|
||||
peaks, duration = self.waveform_generator.generate_peaks(
|
||||
local_audio_path, peaks_per_second=peaks_per_second
|
||||
)
|
||||
|
||||
quantized = np.clip(np.round(peaks * 255.0), 0, 255).astype(np.uint8)
|
||||
payload = {
|
||||
"peaks_b64": base64.b64encode(quantized.tobytes()).decode("ascii"),
|
||||
"encoding": "u8",
|
||||
"peaks_per_second": peaks_per_second,
|
||||
"duration_seconds": duration,
|
||||
# Keyed to the LOGICAL source (the stem path) so a regenerated stem
|
||||
# at a new path invalidates, while the OGG decode shortcut doesn't.
|
||||
"source_gcs_path": gcs_audio_path,
|
||||
}
|
||||
|
||||
try:
|
||||
self.storage_service.upload_json(cache_path, payload)
|
||||
logger.info(f"[{job_id}] Vocals peaks cached at {cache_path}")
|
||||
except Exception as e:
|
||||
logger.warning(f"[{job_id}] Failed to cache vocals peaks (non-fatal): {e}")
|
||||
|
||||
return payload
|
||||
|
||||
def cache_waveform_data(
|
||||
self,
|
||||
gcs_audio_path: str,
|
||||
|
||||
@@ -556,3 +556,105 @@ class TestGetReviewWaveform:
|
||||
)
|
||||
assert amps == [0.1, 0.9]
|
||||
assert duration == 200.0
|
||||
|
||||
|
||||
class TestGetVocalsPeaks:
|
||||
"""Persistent GCS cache of the vocals peak envelope (sub-second strips)."""
|
||||
|
||||
def _service(self, mock_storage_class, mock_waveform_class):
|
||||
import numpy as np
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage_class.return_value = mock_storage
|
||||
mock_waveform = Mock()
|
||||
mock_waveform.generate_peaks.return_value = (
|
||||
np.array([0.0, 0.5, 1.0], dtype=np.float32), 3.0
|
||||
)
|
||||
mock_waveform_class.return_value = mock_waveform
|
||||
return AudioAnalysisService(), mock_storage, mock_waveform
|
||||
|
||||
@patch("backend.services.audio_analysis_service.StorageService")
|
||||
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
|
||||
@patch("backend.services.audio_analysis_service.WaveformGenerator")
|
||||
def test_cache_hit_skips_decode(
|
||||
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
|
||||
):
|
||||
service, storage, waveform = self._service(mock_storage_class, mock_waveform_class)
|
||||
cached = {
|
||||
"peaks_b64": "AAA=",
|
||||
"encoding": "u8",
|
||||
"peaks_per_second": 400,
|
||||
"duration_seconds": 3.0,
|
||||
"source_gcs_path": "jobs/j1/stems/vocals_clean.flac",
|
||||
}
|
||||
storage.file_exists.return_value = True
|
||||
storage.download_json.return_value = cached
|
||||
|
||||
result = service.get_vocals_peaks("jobs/j1/stems/vocals_clean.flac", "j1", 400)
|
||||
|
||||
assert result == cached
|
||||
waveform.generate_peaks.assert_not_called()
|
||||
storage.upload_json.assert_not_called()
|
||||
|
||||
@patch("backend.services.audio_analysis_service.StorageService")
|
||||
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
|
||||
@patch("backend.services.audio_analysis_service.WaveformGenerator")
|
||||
def test_cache_miss_computes_quantizes_and_uploads(
|
||||
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
|
||||
):
|
||||
import base64
|
||||
|
||||
service, storage, waveform = self._service(mock_storage_class, mock_waveform_class)
|
||||
storage.file_exists.return_value = False
|
||||
|
||||
result = service.get_vocals_peaks("jobs/j1/stems/vocals_clean.flac", "j1", 400)
|
||||
|
||||
assert result["encoding"] == "u8"
|
||||
assert result["peaks_per_second"] == 400
|
||||
assert result["duration_seconds"] == 3.0
|
||||
# [0.0, 0.5, 1.0] -> uint8 [0, 128, 255]
|
||||
assert list(base64.b64decode(result["peaks_b64"])) == [0, 128, 255]
|
||||
cache_path, payload = storage.upload_json.call_args[0]
|
||||
assert cache_path == "jobs/j1/review-audio/vocals_peaks_400.json"
|
||||
assert payload == result
|
||||
|
||||
@patch("backend.services.audio_analysis_service.StorageService")
|
||||
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
|
||||
@patch("backend.services.audio_analysis_service.WaveformGenerator")
|
||||
def test_stale_source_recomputes(
|
||||
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
|
||||
):
|
||||
service, storage, waveform = self._service(mock_storage_class, mock_waveform_class)
|
||||
storage.file_exists.return_value = True
|
||||
storage.download_json.return_value = {
|
||||
"peaks_b64": "AAA=",
|
||||
"peaks_per_second": 400,
|
||||
"duration_seconds": 3.0,
|
||||
"source_gcs_path": "jobs/j1/stems/OLD.flac",
|
||||
}
|
||||
|
||||
service.get_vocals_peaks("jobs/j1/stems/vocals_clean.flac", "j1", 400)
|
||||
waveform.generate_peaks.assert_called_once()
|
||||
|
||||
@patch("backend.services.audio_analysis_service.StorageService")
|
||||
@patch("backend.services.audio_analysis_service.AudioAnalyzer")
|
||||
@patch("backend.services.audio_analysis_service.WaveformGenerator")
|
||||
def test_prefers_transcoded_ogg_and_survives_upload_failure(
|
||||
self, mock_waveform_class, mock_analyzer_class, mock_storage_class
|
||||
):
|
||||
service, storage, waveform = self._service(mock_storage_class, mock_waveform_class)
|
||||
storage.file_exists.side_effect = lambda p: p.endswith(".ogg")
|
||||
storage.upload_json.side_effect = Exception("GCS write denied")
|
||||
|
||||
transcoding = Mock()
|
||||
transcoding.get_transcoded_cache_path.return_value = "jobs/j1/review-audio/vocals_clean.ogg"
|
||||
|
||||
result = service.get_vocals_peaks(
|
||||
"jobs/j1/stems/vocals_clean.flac", "j1", 400, transcoding
|
||||
)
|
||||
|
||||
# Decoded from the small OGG, and data still returned despite upload failure
|
||||
downloaded_path = storage.download_file.call_args[0][0]
|
||||
assert downloaded_path == "jobs/j1/review-audio/vocals_clean.ogg"
|
||||
assert result["source_gcs_path"] == "jobs/j1/stems/vocals_clean.flac"
|
||||
@@ -2139,3 +2139,75 @@ class TestLocalPreviewRenderBound:
|
||||
assert await review_module._render_preview_locally(MagicMock(), {}, False) == "hash2"
|
||||
|
||||
asyncio.run(scenario())
|
||||
|
||||
|
||||
class TestVocalsPeaksEndpoint:
|
||||
"""/vocals-peaks serves the precomputed envelope (sub-second strips)."""
|
||||
|
||||
def _run(self, stems=None, audio_complete=True, pps=400):
|
||||
import asyncio
|
||||
from backend.api.routes.review import get_vocals_peaks
|
||||
|
||||
job = MagicMock()
|
||||
job.input_media_gcs_path = "jobs/j/input.flac"
|
||||
if stems is None:
|
||||
stems = {"vocals_clean": "jobs/j/stems/vocals_clean.flac"}
|
||||
job.file_urls = {"stems": stems}
|
||||
job.state_data = {"audio_complete": audio_complete}
|
||||
|
||||
job_manager = MagicMock()
|
||||
job_manager.get_job.return_value = job
|
||||
|
||||
analysis = MagicMock()
|
||||
analysis.get_vocals_peaks.return_value = {
|
||||
"peaks_b64": "AAECAw==",
|
||||
"encoding": "u8",
|
||||
"peaks_per_second": 400,
|
||||
"duration_seconds": 12.5,
|
||||
"source_gcs_path": "jobs/j/stems/vocals_clean.flac",
|
||||
}
|
||||
|
||||
with patch("backend.api.routes.review.JobManager", return_value=job_manager), patch(
|
||||
"backend.services.audio_analysis_service.AudioAnalysisService",
|
||||
return_value=analysis,
|
||||
), patch(
|
||||
"backend.services.audio_transcoding_service.AudioTranscodingService",
|
||||
return_value=MagicMock(),
|
||||
):
|
||||
resp = asyncio.run(
|
||||
get_vocals_peaks("j", peaks_per_second=pps, auth_info=("admin", "full"))
|
||||
)
|
||||
return resp, analysis
|
||||
|
||||
def test_serves_payload_with_cache_headers(self):
|
||||
import json as jsonlib
|
||||
|
||||
resp, analysis = self._run()
|
||||
|
||||
assert resp.status_code == 200
|
||||
body = jsonlib.loads(resp.body)
|
||||
assert body["peaks_b64"] == "AAECAw=="
|
||||
assert body["encoding"] == "u8"
|
||||
assert body["duration_seconds"] == 12.5
|
||||
assert "max-age" in resp.headers.get("Cache-Control", "")
|
||||
args = analysis.get_vocals_peaks.call_args[0]
|
||||
assert args[0] == "jobs/j/stems/vocals_clean.flac"
|
||||
assert args[1] == "j"
|
||||
assert args[2] == 400
|
||||
|
||||
def test_clamps_peaks_per_second(self):
|
||||
_, analysis = self._run(pps=999999)
|
||||
assert analysis.get_vocals_peaks.call_args[0][2] == 1000
|
||||
|
||||
def test_202_while_separation_running(self):
|
||||
resp, analysis = self._run(stems={}, audio_complete=False)
|
||||
assert resp.status_code == 202
|
||||
assert resp.headers.get("Retry-After") == "15"
|
||||
analysis.get_vocals_peaks.assert_not_called()
|
||||
|
||||
def test_404_when_stem_will_never_exist(self):
|
||||
from fastapi import HTTPException
|
||||
|
||||
with pytest.raises(HTTPException) as exc_info:
|
||||
self._run(stems={}, audio_complete=True)
|
||||
assert exc_info.value.status_code == 404
|
||||
@@ -0,0 +1,20 @@
|
||||
"""Stem-selection helpers shared by review routes and workers."""
|
||||
|
||||
from typing import Optional
|
||||
|
||||
|
||||
def vocals_stem_path(job) -> Optional[str]:
|
||||
"""The full-vocal-mix stem for the review waveform, or None if absent.
|
||||
|
||||
Stem keys vary by separation model. We want the full vocal mix
|
||||
(lead + backing) so every word can be lined up against the waveform:
|
||||
- "vocals" : full vocal from a 2-stem split (rare)
|
||||
- "vocals_clean" : full vocal from the primary vocal/instrumental
|
||||
split (present on essentially all cloud jobs)
|
||||
- "lead_vocals" : last-resort fallback (misses backing lines)
|
||||
"""
|
||||
stems = job.file_urls.get("stems", {}) if job.file_urls else {}
|
||||
for key in ("vocals", "vocals_clean", "lead_vocals"):
|
||||
if stems.get(key):
|
||||
return stems[key]
|
||||
return None
|
||||
@@ -786,6 +786,23 @@ async def _transcode_review_audio(
|
||||
# Non-fatal: /waveform-data computes + caches lazily on first open
|
||||
job_log.warning(f"Review waveform pre-compute failed (non-fatal): {e}")
|
||||
|
||||
# Pre-compute the vocals peak envelope too (Waveforms review mode) so
|
||||
# /vocals-peaks is a pure cache read and the per-segment strips paint
|
||||
# sub-second on first open.
|
||||
from backend.utils.stems import vocals_stem_path
|
||||
|
||||
vocals_source = vocals_stem_path(job)
|
||||
if vocals_source:
|
||||
try:
|
||||
await asyncio.to_thread(
|
||||
AudioAnalysisService().get_vocals_peaks,
|
||||
vocals_source, job_id, 400, transcoding,
|
||||
)
|
||||
job_log.info("Vocals peak envelope pre-computed")
|
||||
except Exception as e:
|
||||
# Non-fatal: /vocals-peaks computes + caches lazily on first open
|
||||
job_log.warning(f"Vocals peaks pre-compute failed (non-fatal): {e}")
|
||||
|
||||
except Exception as e:
|
||||
# Non-fatal: review UI will fall back to FLAC signed URLs
|
||||
job_log.warning(f"Review audio transcoding failed (non-fatal): {e}")
|
||||
|
||||
+17
@@ -2696,3 +2696,20 @@ localhost no-op). Added in v0.231.0 — see
|
||||
`docs/archive/2026-09-18-concurrent-review-reliability-investigation-and-plan.md`.
|
||||
|
||||
Crash reports are the separate `POST /api/client-errors` (error-monitor pipeline).
|
||||
|
||||
### Vocals Peaks (Waveforms review mode)
|
||||
|
||||
```http
|
||||
GET /api/review/{job_id}/vocals-peaks?peaks_per_second=400
|
||||
```
|
||||
|
||||
Pre-computed max-abs peak envelope of the vocal stem:
|
||||
`{peaks_b64, encoding: "u8", peaks_per_second, duration_seconds}` — base64 of one
|
||||
uint8 per 1/400s bucket (~150 KB for a 5-minute track). Replaces the frontend's
|
||||
download+decode of the whole vocals OGG so Waveforms-mode strips paint sub-second.
|
||||
Served from a persistent GCS cache (`review-audio/vocals_peaks_{pps}.json`,
|
||||
validated against the stem path), pre-computed by `screens_worker`; a miss computes
|
||||
off the event loop under the shared waveform semaphore. Returns `202 Retry-After: 15`
|
||||
while audio separation is still producing the stem. Auth like `/audio/vocals`
|
||||
(`?token=` / `?review_token=`). The frontend falls back to the full audio decode if
|
||||
this endpoint fails (`VocalsAudioDataLoader`). Added in v0.232.0.
|
||||
@@ -88,6 +88,9 @@ interface ApiClient {
|
||||
searchLyrics?: (artist: string, title: string, forceSources?: string[]) => Promise<SearchLyricsResponse>
|
||||
getAudioUrl: (hash: string) => string
|
||||
getVocalsAudioUrl: () => string
|
||||
// Optional: pre-computed peaks endpoint (cloud mode); absent → loader falls
|
||||
// back to downloading + decoding the full stem.
|
||||
getVocalsPeaksUrl?: () => string
|
||||
generatePreviewVideo: (data: CorrectionData, isDuet?: boolean) => Promise<{
|
||||
status: string
|
||||
message?: string
|
||||
@@ -177,6 +180,7 @@ export default function LyricsAnalyzer({
|
||||
const [instrumentalChoice, setInstrumentalChoice] = useState<'clean' | 'with_backing' | null>(null)
|
||||
|
||||
const [vocalsAudioUrl, setVocalsAudioUrl] = useState<string | null>(null)
|
||||
const [vocalsPeaksUrl, setVocalsPeaksUrl] = useState<string | null>(null)
|
||||
|
||||
// Success screen state (for auto-close after submission)
|
||||
const [showSuccess, setShowSuccess] = useState(false)
|
||||
@@ -657,6 +661,7 @@ export default function LyricsAnalyzer({
|
||||
if (!apiClient) return
|
||||
|
||||
setVocalsAudioUrl(apiClient.getVocalsAudioUrl())
|
||||
setVocalsPeaksUrl(apiClient.getVocalsPeaksUrl ? apiClient.getVocalsPeaksUrl() : null)
|
||||
}, [apiClient])
|
||||
|
||||
// Countdown effect for success screen (auto-close/redirect after submission)
|
||||
@@ -1532,7 +1537,7 @@ export default function LyricsAnalyzer({
|
||||
}
|
||||
|
||||
return (
|
||||
<VocalsAudioDataLoader audioUrl={vocalsAudioUrl}>
|
||||
<VocalsAudioDataLoader audioUrl={vocalsAudioUrl} peaksUrl={vocalsPeaksUrl}>
|
||||
<div className="max-w-full overflow-x-hidden">
|
||||
<Header
|
||||
isReadOnly={isReadOnly}
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { AudioData, AudioNotReadyError, fetchAudioData, isTransientAudioError } from '@/lib/audio-data'
|
||||
import { AudioData, AudioNotReadyError, fetchAudioData, fetchVocalsPeaks, isTransientAudioError } from '@/lib/audio-data'
|
||||
import { reportDegradationEvent } from '@/lib/degradation-events'
|
||||
import { createContext, PropsWithChildren, useEffect, useState } from 'react'
|
||||
|
||||
@@ -11,6 +11,10 @@ export const VocalsAudioDataLoaderContext = createContext<{ audioData: AudioData
|
||||
|
||||
export interface AudioDataLoaderProps extends PropsWithChildren {
|
||||
audioUrl: string | null
|
||||
/** Pre-computed peak-envelope endpoint (~150 KB JSON). Preferred over
|
||||
* downloading + decoding the whole stem from audioUrl; when it fails
|
||||
* terminally the loader falls back to the full audio decode. */
|
||||
peaksUrl?: string | null
|
||||
}
|
||||
|
||||
// Audio separation runs in the background while the user reviews lyrics, so the
|
||||
@@ -25,11 +29,11 @@ const NOT_READY_MAX_RETRIES = 40 // 40 × 15s = 10 minutes
|
||||
// Waveforms view stripless until a manual reload. Retry a few times with backoff.
|
||||
const TRANSIENT_RETRY_DELAYS_MS = [5_000, 15_000, 30_000, 60_000]
|
||||
|
||||
export const VocalsAudioDataLoader = ({ audioUrl, children }: AudioDataLoaderProps) => {
|
||||
export const VocalsAudioDataLoader = ({ audioUrl, peaksUrl, children }: AudioDataLoaderProps) => {
|
||||
const [audioData, setAudioData] = useState<AudioData | null>(null)
|
||||
|
||||
useEffect(() => {
|
||||
if (!audioUrl) return
|
||||
if (!audioUrl && !peaksUrl) return
|
||||
|
||||
// Guard against (a) an unhandled rejection when the endpoint 404s (no vocal
|
||||
// stem for this job) and (b) a stale in-flight fetch resolving after a newer
|
||||
@@ -38,8 +42,10 @@ export const VocalsAudioDataLoader = ({ audioUrl, children }: AudioDataLoaderPro
|
||||
let retryTimer: ReturnType<typeof setTimeout> | undefined
|
||||
const startedAt = Date.now()
|
||||
|
||||
const load = (attempt: number, transientAttempt: number) => {
|
||||
fetchAudioData(audioUrl)
|
||||
const load = (attempt: number, transientAttempt: number, usePeaks: boolean) => {
|
||||
const source =
|
||||
usePeaks && peaksUrl ? fetchVocalsPeaks(peaksUrl) : fetchAudioData(audioUrl!)
|
||||
source
|
||||
.then((audioData) => {
|
||||
if (cancelled) return
|
||||
const elapsedMs = Date.now() - startedAt
|
||||
@@ -48,6 +54,7 @@ export const VocalsAudioDataLoader = ({ audioUrl, children }: AudioDataLoaderPro
|
||||
elapsed_ms: elapsedMs,
|
||||
not_ready_polls: attempt,
|
||||
transient_retries: transientAttempt,
|
||||
used_peaks: usePeaks,
|
||||
})
|
||||
}
|
||||
setAudioData(audioData)
|
||||
@@ -55,13 +62,25 @@ export const VocalsAudioDataLoader = ({ audioUrl, children }: AudioDataLoaderPro
|
||||
.catch((error) => {
|
||||
if (cancelled) return
|
||||
if (error instanceof AudioNotReadyError && attempt < NOT_READY_MAX_RETRIES) {
|
||||
retryTimer = setTimeout(() => load(attempt + 1, transientAttempt), NOT_READY_RETRY_MS)
|
||||
retryTimer = setTimeout(() => load(attempt + 1, transientAttempt, usePeaks), NOT_READY_RETRY_MS)
|
||||
return
|
||||
}
|
||||
if (isTransientAudioError(error) && transientAttempt < TRANSIENT_RETRY_DELAYS_MS.length) {
|
||||
// Peaks get ONE quick retry, then we fall back to the audio path
|
||||
// (which has the full backoff ladder) — a flaky peaks endpoint
|
||||
// must not delay strips by the whole retry ladder first.
|
||||
const maxTransientRetries = usePeaks ? 1 : TRANSIENT_RETRY_DELAYS_MS.length
|
||||
if (isTransientAudioError(error) && transientAttempt < maxTransientRetries) {
|
||||
const delay = TRANSIENT_RETRY_DELAYS_MS[transientAttempt]
|
||||
console.warn(`Vocals audio load failed, retrying in ${delay / 1000}s`, error)
|
||||
retryTimer = setTimeout(() => load(attempt, transientAttempt + 1), delay)
|
||||
console.warn(`Vocals ${usePeaks ? 'peaks' : 'audio'} load failed, retrying in ${delay / 1000}s`, error)
|
||||
retryTimer = setTimeout(() => load(attempt, transientAttempt + 1, usePeaks), delay)
|
||||
return
|
||||
}
|
||||
if (usePeaks && audioUrl) {
|
||||
// Peaks endpoint terminally failed (or exhausted retries) —
|
||||
// fall back to the original full-download-and-decode path so a
|
||||
// peaks-side problem never costs the user their waveforms.
|
||||
console.warn('Vocals peaks unavailable, falling back to full audio decode', error)
|
||||
load(attempt, 0, false)
|
||||
return
|
||||
}
|
||||
console.error('Failed to load vocals audio data', error)
|
||||
@@ -75,14 +94,14 @@ export const VocalsAudioDataLoader = ({ audioUrl, children }: AudioDataLoaderPro
|
||||
})
|
||||
}
|
||||
|
||||
load(0, 0)
|
||||
load(0, 0, Boolean(peaksUrl))
|
||||
|
||||
return () => {
|
||||
cancelled = true
|
||||
if (retryTimer) clearTimeout(retryTimer)
|
||||
setAudioData(null)
|
||||
}
|
||||
}, [audioUrl])
|
||||
}, [audioUrl, peaksUrl])
|
||||
|
||||
return (
|
||||
<VocalsAudioDataLoaderContext.Provider value={{ audioData }}>
|
||||
|
||||
@@ -38,6 +38,9 @@ function makeMockApiClient(): LyricsReviewApiClient {
|
||||
searchLyrics: jest.fn(),
|
||||
getAudioUrl: jest.fn(),
|
||||
getVocalsAudioUrl: jest.fn(),
|
||||
getVocalsPeaksUrl: jest.fn(),
|
||||
getWaveformData: jest.fn(),
|
||||
refreshInstrumentalUrls: jest.fn(),
|
||||
generatePreviewVideo: jest.fn(),
|
||||
getPreviewVideoStatus: jest.fn(),
|
||||
getPreviewVideoUrl: jest.fn(),
|
||||
|
||||
@@ -5,7 +5,7 @@ import { VocalsAudioDataLoader, VocalsAudioDataLoaderContext } from '../VocalsAu
|
||||
|
||||
jest.mock('@/lib/audio-data', () => {
|
||||
const actual = jest.requireActual('@/lib/audio-data')
|
||||
return { ...actual, fetchAudioData: jest.fn() }
|
||||
return { ...actual, fetchAudioData: jest.fn(), fetchVocalsPeaks: jest.fn() }
|
||||
})
|
||||
|
||||
const fetchAudioData = fetchAudioDataImport as jest.Mock
|
||||
@@ -185,3 +185,108 @@ describe('VocalsAudioDataLoader', () => {
|
||||
expect(fetchAudioData).toHaveBeenCalledTimes(1)
|
||||
})
|
||||
})
|
||||
|
||||
describe('VocalsAudioDataLoader peaks-first behavior', () => {
|
||||
const audioDataLib = jest.requireMock('@/lib/audio-data')
|
||||
|
||||
beforeEach(() => {
|
||||
jest.clearAllMocks()
|
||||
jest.useFakeTimers()
|
||||
audioDataLib.fetchVocalsPeaks = jest.fn()
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
jest.useRealTimers()
|
||||
})
|
||||
|
||||
it('prefers the peaks endpoint and never downloads the full audio', async () => {
|
||||
audioDataLib.fetchVocalsPeaks.mockResolvedValue(AUDIO_DATA)
|
||||
|
||||
render(
|
||||
<VocalsAudioDataLoader audioUrl="https://api/audio/vocals" peaksUrl="https://api/vocals-peaks">
|
||||
<Probe />
|
||||
</VocalsAudioDataLoader>
|
||||
)
|
||||
await flushPromises()
|
||||
|
||||
expect(screen.getByTestId('probe')).toHaveTextContent('loaded')
|
||||
expect(audioDataLib.fetchVocalsPeaks).toHaveBeenCalledTimes(1)
|
||||
expect(fetchAudioData).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('falls back to full audio decode when peaks fail terminally', async () => {
|
||||
audioDataLib.fetchVocalsPeaks.mockRejectedValue(new AudioFetchError(404))
|
||||
fetchAudioData.mockResolvedValue(AUDIO_DATA)
|
||||
const consoleWarn = jest.spyOn(console, 'warn').mockImplementation(() => {})
|
||||
|
||||
render(
|
||||
<VocalsAudioDataLoader audioUrl="https://api/audio/vocals" peaksUrl="https://api/vocals-peaks">
|
||||
<Probe />
|
||||
</VocalsAudioDataLoader>
|
||||
)
|
||||
await flushPromises()
|
||||
|
||||
expect(fetchAudioData).toHaveBeenCalledTimes(1)
|
||||
expect(screen.getByTestId('probe')).toHaveTextContent('loaded')
|
||||
consoleWarn.mockRestore()
|
||||
})
|
||||
|
||||
it('retries peaks once on a transient error, then falls back to audio', async () => {
|
||||
audioDataLib.fetchVocalsPeaks.mockRejectedValue(new AudioFetchError(503))
|
||||
fetchAudioData.mockResolvedValue(AUDIO_DATA)
|
||||
const consoleWarn = jest.spyOn(console, 'warn').mockImplementation(() => {})
|
||||
|
||||
render(
|
||||
<VocalsAudioDataLoader audioUrl="https://api/audio/vocals" peaksUrl="https://api/vocals-peaks">
|
||||
<Probe />
|
||||
</VocalsAudioDataLoader>
|
||||
)
|
||||
await flushPromises()
|
||||
expect(audioDataLib.fetchVocalsPeaks).toHaveBeenCalledTimes(1)
|
||||
|
||||
// One quick retry of peaks...
|
||||
await act(async () => {
|
||||
jest.advanceTimersByTime(5_000)
|
||||
})
|
||||
expect(audioDataLib.fetchVocalsPeaks).toHaveBeenCalledTimes(2)
|
||||
// ...then immediate fallback to the audio path (no more peaks calls).
|
||||
expect(fetchAudioData).toHaveBeenCalledTimes(1)
|
||||
expect(screen.getByTestId('probe')).toHaveTextContent('loaded')
|
||||
|
||||
consoleWarn.mockRestore()
|
||||
})
|
||||
|
||||
it('polls peaks on 202 while separation is running', async () => {
|
||||
audioDataLib.fetchVocalsPeaks
|
||||
.mockRejectedValueOnce(new AudioNotReadyError())
|
||||
.mockResolvedValue(AUDIO_DATA)
|
||||
|
||||
render(
|
||||
<VocalsAudioDataLoader audioUrl="https://api/audio/vocals" peaksUrl="https://api/vocals-peaks">
|
||||
<Probe />
|
||||
</VocalsAudioDataLoader>
|
||||
)
|
||||
await flushPromises()
|
||||
expect(screen.getByTestId('probe')).toHaveTextContent('empty')
|
||||
|
||||
await act(async () => {
|
||||
jest.advanceTimersByTime(15_000)
|
||||
})
|
||||
expect(screen.getByTestId('probe')).toHaveTextContent('loaded')
|
||||
expect(fetchAudioData).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('works with only audioUrl (local mode without a peaks endpoint)', async () => {
|
||||
fetchAudioData.mockResolvedValue(AUDIO_DATA)
|
||||
|
||||
render(
|
||||
<VocalsAudioDataLoader audioUrl="https://api/audio/vocals">
|
||||
<Probe />
|
||||
</VocalsAudioDataLoader>
|
||||
)
|
||||
await flushPromises()
|
||||
|
||||
expect(screen.getByTestId('probe')).toHaveTextContent('loaded')
|
||||
expect(audioDataLib.fetchVocalsPeaks).not.toHaveBeenCalled()
|
||||
})
|
||||
})
|
||||
@@ -127,4 +127,32 @@ test.describe('Review fast-full-load contract', () => {
|
||||
expect(data.amplitudes.length).toBeGreaterThan(0);
|
||||
expect(data.duration_seconds).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
// Sub-second strips (v0.232.0): precomputed vocals peak envelope.
|
||||
test('vocals-peaks returns a compact cached envelope quickly', async ({ request }) => {
|
||||
const base = `${API_URL}/api/review/${JOB_ID}/vocals-peaks`;
|
||||
const url = REVIEW_TOKEN ? `${base}?review_token=${encodeURIComponent(REVIEW_TOKEN)}` : base;
|
||||
|
||||
// First call may compute+cache; the second must be a pure cache read.
|
||||
const warm = await request.get(url, { headers: authHeaders() });
|
||||
expect([200, 202]).toContain(warm.status());
|
||||
if (warm.status() === 202) return; // separation still running for this job
|
||||
|
||||
const started = Date.now();
|
||||
const res = await request.get(url, { headers: authHeaders() });
|
||||
const elapsedMs = Date.now() - started;
|
||||
console.log(`vocals-peaks (cached) status=${res.status()} elapsed=${elapsedMs}ms`);
|
||||
expect(res.ok()).toBeTruthy();
|
||||
expect(elapsedMs).toBeLessThan(5_000);
|
||||
expect(res.headers()['cache-control']).toContain('max-age');
|
||||
|
||||
const data = await res.json();
|
||||
expect(data.encoding).toBe('u8');
|
||||
expect(typeof data.peaks_b64).toBe('string');
|
||||
expect(data.peaks_b64.length).toBeGreaterThan(0);
|
||||
expect(data.peaks_per_second).toBe(400);
|
||||
expect(data.duration_seconds).toBeGreaterThan(0);
|
||||
// Envelope should be dramatically smaller than the audio it replaces.
|
||||
expect(data.peaks_b64.length).toBeLessThan(1_000_000);
|
||||
});
|
||||
});
|
||||
@@ -116,3 +116,47 @@ describe('computePeaks', () => {
|
||||
expect(peaks[0]).toBeCloseTo(0.5)
|
||||
})
|
||||
})
|
||||
|
||||
describe('fetchVocalsPeaks', () => {
|
||||
const { fetchVocalsPeaks, AudioNotReadyError, AudioFetchError } = jest.requireActual('@/lib/audio-data')
|
||||
|
||||
const mockJsonResponse = (status: number, body?: unknown) =>
|
||||
({ ok: status >= 200 && status < 300, status, json: async () => body }) as Response
|
||||
|
||||
afterEach(() => {
|
||||
jest.restoreAllMocks()
|
||||
})
|
||||
|
||||
it('decodes the u8/base64 payload into AudioData', async () => {
|
||||
// peaks [0, 128, 255] -> [0, ~0.5, 1]
|
||||
const b64 = btoa(String.fromCharCode(0, 128, 255))
|
||||
global.fetch = jest.fn(async () =>
|
||||
mockJsonResponse(200, {
|
||||
peaks_b64: b64,
|
||||
encoding: 'u8',
|
||||
peaks_per_second: 400,
|
||||
duration_seconds: 0.0075,
|
||||
})
|
||||
) as unknown as typeof fetch
|
||||
|
||||
const data = await fetchVocalsPeaks('https://api/vocals-peaks')
|
||||
expect(data.peaksPerSecond).toBe(400)
|
||||
expect(data.duration).toBeCloseTo(0.0075)
|
||||
expect(Array.from(data.peaks)).toEqual([0, Math.fround(128 / 255), 1])
|
||||
})
|
||||
|
||||
it('throws AudioNotReadyError on 202', async () => {
|
||||
global.fetch = jest.fn(async () => mockJsonResponse(202)) as unknown as typeof fetch
|
||||
await expect(fetchVocalsPeaks('u')).rejects.toBeInstanceOf(AudioNotReadyError)
|
||||
})
|
||||
|
||||
it('throws AudioFetchError with status on non-2xx', async () => {
|
||||
global.fetch = jest.fn(async () => mockJsonResponse(404)) as unknown as typeof fetch
|
||||
await expect(fetchVocalsPeaks('u')).rejects.toMatchObject({ status: 404 })
|
||||
})
|
||||
|
||||
it('treats a malformed payload as a transient server error', async () => {
|
||||
global.fetch = jest.fn(async () => mockJsonResponse(200, { nope: true })) as unknown as typeof fetch
|
||||
await expect(fetchVocalsPeaks('u')).rejects.toBeInstanceOf(AudioFetchError)
|
||||
})
|
||||
})
|
||||
@@ -3921,6 +3921,7 @@ export interface LyricsReviewApiClient {
|
||||
searchLyrics: (artist: string, title: string, forceSources?: string[]) => Promise<SearchLyricsResponse>
|
||||
getAudioUrl: (hash: string) => string
|
||||
getVocalsAudioUrl: () => string
|
||||
getVocalsPeaksUrl: () => string
|
||||
generatePreviewVideo: (data: CorrectionData, isDuet?: boolean) => Promise<{
|
||||
status: string
|
||||
message?: string
|
||||
@@ -4075,6 +4076,16 @@ export function createLyricsReviewApiClient(jobId: string): LyricsReviewApiClien
|
||||
return token ? `${base}?token=${encodeURIComponent(token)}` : base
|
||||
},
|
||||
|
||||
/**
|
||||
* Pre-computed vocals peak-envelope URL (Waveforms review mode).
|
||||
* ~150 KB JSON instead of downloading + decoding the whole vocal stem.
|
||||
*/
|
||||
getVocalsPeaksUrl(): string {
|
||||
const token = getAccessToken()
|
||||
const base = `${API_BASE_URL}/api/review/${jobId}/vocals-peaks`
|
||||
return token ? `${base}?token=${encodeURIComponent(token)}` : base
|
||||
},
|
||||
|
||||
/**
|
||||
* Generate a preview video from the current correction data.
|
||||
*
|
||||
|
||||
@@ -93,6 +93,37 @@ export function computePeaks(decoded: DecodedAudioLike): AudioData {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetch the server-side pre-computed vocals peak envelope (~150 KB JSON) and
|
||||
* adapt it to the same AudioData shape computePeaks produces — so waveform
|
||||
* strips paint sub-second instead of after a multi-MB download + decode.
|
||||
* Throws AudioNotReadyError on 202 (separation still running) and
|
||||
* AudioFetchError on other non-2xx, mirroring fetchAudioData.
|
||||
*/
|
||||
export async function fetchVocalsPeaks(url: string): Promise<AudioData> {
|
||||
const response = await fetch(url)
|
||||
if (response.status === 202) {
|
||||
throw new AudioNotReadyError()
|
||||
}
|
||||
if (!response.ok) {
|
||||
throw new AudioFetchError(response.status)
|
||||
}
|
||||
const data = await response.json()
|
||||
if (data?.encoding !== 'u8' || typeof data.peaks_b64 !== 'string' || !data.peaks_b64) {
|
||||
throw new AudioFetchError(500) // unexpected payload — treat as transient server trouble
|
||||
}
|
||||
const raw = atob(data.peaks_b64)
|
||||
const peaks = new Float32Array(raw.length)
|
||||
for (let i = 0; i < raw.length; i++) {
|
||||
peaks[i] = raw.charCodeAt(i) / 255
|
||||
}
|
||||
return {
|
||||
duration: Number(data.duration_seconds) || 0,
|
||||
peaks,
|
||||
peaksPerSecond: Number(data.peaks_per_second) || PEAKS_PER_SECOND,
|
||||
}
|
||||
}
|
||||
|
||||
export async function fetchAudioData(url: string): Promise<AudioData> {
|
||||
// fetch() resolves even for 4xx/5xx; the vocals endpoint 404s when a job has
|
||||
// no vocal stem. Fail explicitly so the caller sees "no vocals" rather than an
|
||||
|
||||
@@ -208,7 +208,60 @@ class WaveformGenerator:
|
||||
amplitudes.append(normalized)
|
||||
|
||||
return amplitudes, duration_seconds
|
||||
|
||||
|
||||
def generate_peaks(
|
||||
self,
|
||||
audio_path: str,
|
||||
peaks_per_second: int = 400,
|
||||
) -> Tuple[np.ndarray, float]:
|
||||
"""
|
||||
Generate a max-abs peak envelope at a fixed time resolution.
|
||||
|
||||
Mirrors the review frontend's ``computePeaks`` (lib/audio-data.ts):
|
||||
one peak per 1/peaks_per_second bucket, each the maximum absolute
|
||||
sample value across all channels, normalized to 0.0–1.0. Serving this
|
||||
pre-computed lets the review page draw per-segment waveform strips
|
||||
without downloading + decoding the whole vocal stem client-side.
|
||||
|
||||
Vectorized (numpy) — a 4-minute stereo track computes in well under a
|
||||
second, vs the pure-Python RMS loop in generate_data_only.
|
||||
|
||||
Returns:
|
||||
Tuple of (peaks float32 ndarray in 0..1, duration_seconds)
|
||||
"""
|
||||
path = Path(audio_path)
|
||||
if not path.exists():
|
||||
raise FileNotFoundError(f"Audio file not found: {audio_path}")
|
||||
|
||||
audio = AudioSegment.from_file(audio_path)
|
||||
duration_seconds = len(audio) / 1000.0
|
||||
|
||||
samples = np.abs(
|
||||
np.asarray(audio.get_array_of_samples(), dtype=np.float32)
|
||||
) / float(audio.max_possible_amplitude)
|
||||
if audio.channels > 1:
|
||||
# Interleaved -> (frames, channels) -> per-frame max across channels
|
||||
frames = len(samples) // audio.channels
|
||||
samples = samples[: frames * audio.channels]
|
||||
samples = samples.reshape(frames, audio.channels).max(axis=1)
|
||||
|
||||
bucket_count = max(1, math.ceil(duration_seconds * peaks_per_second))
|
||||
# Fractional stride with floored boundaries, matching the frontend's
|
||||
# drift-free bucketing (see the 44.1kHz drift fix notes there).
|
||||
samples_per_bucket = len(samples) / bucket_count
|
||||
boundaries = np.floor(np.arange(bucket_count + 1) * samples_per_bucket).astype(np.int64)
|
||||
boundaries[-1] = len(samples)
|
||||
|
||||
peaks = np.zeros(bucket_count, dtype=np.float32)
|
||||
# maximum.reduceat needs strictly valid starts; guard empty buckets.
|
||||
starts = boundaries[:-1]
|
||||
if len(samples):
|
||||
reduced = np.maximum.reduceat(samples, np.minimum(starts, len(samples) - 1))
|
||||
widths = boundaries[1:] - starts
|
||||
peaks = np.where(widths > 0, reduced, 0.0).astype(np.float32)
|
||||
|
||||
return peaks, duration_seconds
|
||||
|
||||
def _get_envelope(
|
||||
self,
|
||||
audio: AudioSegment,
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[tool.poetry]
|
||||
name = "karaoke-gen"
|
||||
version = "0.231.1"
|
||||
version = "0.232.0"
|
||||
description = "Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens."
|
||||
authors = ["Andrew Beveridge <andrew@beveridge.uk>"]
|
||||
license = "MIT"
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
"""Tests for WaveformGenerator.generate_peaks (review Waveforms-mode envelope)."""
|
||||
|
||||
import math
|
||||
import tempfile
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
from pydub import AudioSegment
|
||||
from pydub.generators import Sine
|
||||
|
||||
from karaoke_gen.instrumental_review.waveform import WaveformGenerator
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def stereo_wav(tmp_path_factory):
|
||||
"""3s stereo: -6dB 440Hz left, -20dB 880Hz right."""
|
||||
left = Sine(440).to_audio_segment(duration=3000).apply_gain(-6)
|
||||
right = Sine(880).to_audio_segment(duration=3000).apply_gain(-20)
|
||||
seg = AudioSegment.from_mono_audiosegments(left, right)
|
||||
path = tmp_path_factory.mktemp("audio") / "stereo.wav"
|
||||
seg.export(str(path), format="wav")
|
||||
return str(path)
|
||||
|
||||
|
||||
class TestGeneratePeaks:
|
||||
def test_bucket_count_and_duration(self, stereo_wav):
|
||||
peaks, duration = WaveformGenerator().generate_peaks(stereo_wav, peaks_per_second=400)
|
||||
assert duration == pytest.approx(3.0, abs=0.05)
|
||||
assert len(peaks) == math.ceil(duration * 400)
|
||||
|
||||
def test_peaks_are_max_abs_across_channels_normalized(self, stereo_wav):
|
||||
peaks, _ = WaveformGenerator().generate_peaks(stereo_wav, peaks_per_second=400)
|
||||
# The louder channel (-6 dB sine) dominates: peak amplitude ~10^(-6/20)=0.501
|
||||
assert float(peaks.max()) == pytest.approx(0.501, abs=0.01)
|
||||
assert float(peaks.min()) >= 0.0
|
||||
assert peaks.dtype == np.float32
|
||||
|
||||
def test_matches_naive_reference(self, stereo_wav):
|
||||
pps = 400
|
||||
peaks, duration = WaveformGenerator().generate_peaks(stereo_wav, peaks_per_second=pps)
|
||||
|
||||
audio = AudioSegment.from_file(stereo_wav)
|
||||
s = np.abs(np.asarray(audio.get_array_of_samples(), dtype=np.float32)) / audio.max_possible_amplitude
|
||||
frames = len(s) // audio.channels
|
||||
s = s[: frames * audio.channels].reshape(frames, audio.channels).max(axis=1)
|
||||
bucket_count = math.ceil(duration * pps)
|
||||
spb = len(s) / bucket_count
|
||||
ref = np.array(
|
||||
[
|
||||
s[math.floor(i * spb): (len(s) if i == bucket_count - 1 else max(math.floor((i + 1) * spb), math.floor(i * spb) + 1))].max()
|
||||
for i in range(bucket_count)
|
||||
],
|
||||
dtype=np.float32,
|
||||
)
|
||||
assert np.abs(ref - peaks).max() == 0.0
|
||||
|
||||
def test_missing_file_raises(self):
|
||||
with pytest.raises(FileNotFoundError):
|
||||
WaveformGenerator().generate_peaks("/nonexistent.wav")
|
||||
Reference in new issue
Block a user