feat: Omnizart MIDI transcription — phases 1-3
Phase 1 (omnizart-music MVP): - config.py: add omnizart-music/vocal/vocal-contour to TRANSCRIBE_MODES with engine + description fields; add OMNIZART_ENABLED and OMNIZART_SIDECAR_URL env-var config constants - transcriber_service.py: dispatch run_transcription() to basic-pitch or omnizart based on mode prefix; extract _run_basic_pitch() helper - routers/transcribe.py: add 503 guard for omnizart modes when disabled; GET /modes now returns available flag per mode Phase 2 (all three Omnizart modes): - backend/services/omnizart_service.py (new): WAV conversion via ffmpeg with 10 s minimum padding; subprocess invocation with cancellation polling; vocal mode CWD quirk handled by setting cwd=tmp; all three modes supported Phase 3 (frontend polish + sidecar): - types.ts: extend ModeOption with engine, available, description fields - ProjectDetail.tsx: replace static TRANSCRIBE_MODES with live useQuery call to GET /api/transcribe/modes; add TranscribeModeSelect component with <optgroup> grouping by engine, disabled options for unavailable modes, and title= tooltips from description field; handleTranscribeAll auto-selects omnizart-music when available, falls back to polyphonic - omnizart-sidecar/app.py + Dockerfile (new): minimal FastAPI HTTP wrapper around the Omnizart CLI; single POST /transcribe endpoint; GET /health - stemengine.yml: add omnizart-sidecar service (profiles: [omnizart]); add OMNIZART_ENABLED and OMNIZART_SIDECAR_URL env vars to stemengine service
This commit is contained in:
1 parent
e5ca71f7c4
commit
5c2152c57c
11 files changed
+1152
-34
No files matched your search
@@ -0,0 +1,499 @@
|
||||
# Omnizart MIDI Transcription Integration Plan
|
||||
|
||||
**Date:** 2026-03-14
|
||||
**Status:** Planning
|
||||
**Scope:** Add Omnizart as a second transcription engine alongside basic-pitch.
|
||||
|
||||
---
|
||||
|
||||
## Background
|
||||
|
||||
StemEngine currently offers MIDI transcription via **basic-pitch** (Spotify/Google,
|
||||
in-process Python API) with two modes:
|
||||
|
||||
| ID | Label | Engine |
|
||||
|----|-------|--------|
|
||||
| `polyphonic` | Polyphonic | basic-pitch |
|
||||
| `melody` | Melody / Vocal | basic-pitch |
|
||||
|
||||
**Omnizart** (Music AI NTU, 2021) is a separate toolkit for music transcription. It
|
||||
operates via a CLI (`omnizart {mode} transcribe ...`) and produces standard `.mid`
|
||||
output. A reference integration already exists in the workspace at
|
||||
`docker/x/Omnizart-UI/` (Sanic-based standalone API). The reference implementation
|
||||
was the primary source for this plan.
|
||||
|
||||
### Why add it?
|
||||
|
||||
- basic-pitch is excellent for melody extraction but struggles with polyphonic
|
||||
arrangements at instrument level (piano-roll style).
|
||||
- Omnizart's `music` mode does instrument-aware multi-track transcription — it is
|
||||
the best freely available option for generating separated MIDI per instrument.
|
||||
- The `vocal` mode produces a melody MIDI line comparable to basic-pitch's `melody`
|
||||
mode but via a different model trained specifically on vocal contours.
|
||||
- The `transcribe.py` router already has a `_STEM_PREFERENCE` list ordered for
|
||||
Omnizart's `music` mode (pitched instruments first, drums excluded) — the
|
||||
groundwork is already there.
|
||||
|
||||
---
|
||||
|
||||
## Omnizart Mode Inventory
|
||||
|
||||
| Omnizart mode | CLI arg | Output | Status |
|
||||
|---|---|---|---|
|
||||
| `music` | `omnizart music transcribe -o <out.mid> <in.wav>` | Multi-track MIDI | ✅ Works |
|
||||
| `vocal` | `omnizart vocal transcribe <in.wav>` | Single-track melody MIDI | ✅ Works (quirky `-o` behaviour, see below) |
|
||||
| `vocal-contour` | `omnizart vocal-contour transcribe -o <out.mid> <in.wav>` | Contour F0 MIDI | ✅ Works |
|
||||
| `chord` | — | Chord symbols | ❌ Broken (numpy/Python conflict) |
|
||||
| `drums` | — | Drum MIDI | ❌ Broken (Python 2.7 dep) |
|
||||
|
||||
**Vocal mode output path quirk:** The `-o` flag is silently ignored. The output file
|
||||
is always written to the current working directory as
|
||||
`./{input-stem}-vocal.mid`. This must be handled explicitly (see service design).
|
||||
|
||||
**Minimum file length:** Omnizart silently fails or produces empty output for inputs
|
||||
shorter than ~10 seconds. The reference implementation enforces `10000 ms` as a
|
||||
hard minimum via `SoundUtil.ConvertFileToWav(src, dst, 10_000)`. We must do the same.
|
||||
|
||||
---
|
||||
|
||||
## New Mode IDs for StemEngine
|
||||
|
||||
Three new mode IDs are proposed, prefixed with `omnizart-` to make the dispatch
|
||||
branch unambiguous:
|
||||
|
||||
| New ID | Label | Omnizart CLI mode |
|
||||
|---|---|---|
|
||||
| `omnizart-music` | Omnizart — Multi-instrument | `music` |
|
||||
| `omnizart-vocal` | Omnizart — Vocal Melody | `vocal` |
|
||||
| `omnizart-vocal-contour` | Omnizart — Vocal Contour | `vocal-contour` |
|
||||
|
||||
Phase 1 ships `omnizart-music` only. Phases 2 and 3 add the remaining modes.
|
||||
|
||||
---
|
||||
|
||||
## Files to Change
|
||||
|
||||
### 1. `backend/config.py`
|
||||
|
||||
**Location:** line 195 (`TRANSCRIBE_MODES` list).
|
||||
|
||||
Extend the list with the new modes and add an availability flag so the frontend
|
||||
can grey them out if Omnizart is not installed/reachable:
|
||||
|
||||
```python
|
||||
# basic-pitch transcription modes
|
||||
TRANSCRIBE_MODES = [
|
||||
{"id": "polyphonic", "label": "Polyphonic", "engine": "basic-pitch"},
|
||||
{"id": "melody", "label": "Melody / Vocal", "engine": "basic-pitch"},
|
||||
# Omnizart modes — require Omnizart to be installed (see OMNIZART_ENABLED)
|
||||
{"id": "omnizart-music", "label": "Omnizart — Multi-instrument", "engine": "omnizart"},
|
||||
{"id": "omnizart-vocal", "label": "Omnizart — Vocal Melody", "engine": "omnizart"},
|
||||
{"id": "omnizart-vocal-contour", "label": "Omnizart — Vocal Contour", "engine": "omnizart"},
|
||||
]
|
||||
DEFAULT_TRANSCRIBE_MODE = "polyphonic"
|
||||
|
||||
# Set via env var; when False the Omnizart modes are still listed but the
|
||||
# router returns 503 if one is selected (allows UI to warn without hiding them).
|
||||
OMNIZART_ENABLED = os.environ.get("OMNIZART_ENABLED", "false").lower() == "true"
|
||||
```
|
||||
|
||||
No changes needed to `TRANSCRIBE_PRESETS` — those are basic-pitch only and
|
||||
dispatched conditionally (see service below).
|
||||
|
||||
---
|
||||
|
||||
### 2. `backend/services/omnizart_service.py` *(new file)*
|
||||
|
||||
This service owns all Omnizart-specific logic: WAV conversion, minimum-length
|
||||
padding, subprocess invocation, output path resolution, and cleanup.
|
||||
|
||||
```python
|
||||
"""
|
||||
MIDI transcription via Omnizart CLI (subprocess).
|
||||
Omnizart must be available on PATH (or OMNIZART_BIN env var).
|
||||
Supports modes: music | vocal | vocal-contour
|
||||
"""
|
||||
import logging
|
||||
import os
|
||||
import shlex
|
||||
import subprocess
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
from .. import job_manager as jm
|
||||
from ..config import JOBS_DIR
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
# CLI binary name / path — override with OMNIZART_BIN env var for venv/conda
|
||||
OMNIZART_BIN = os.environ.get("OMNIZART_BIN", "omnizart")
|
||||
|
||||
# Omnizart fails silently on inputs shorter than ~10 s; pad/reject at this limit
|
||||
MIN_DURATION_MS = 10_000
|
||||
|
||||
|
||||
def _omnizart_mode(mode_id: str) -> str:
|
||||
"""Strip the 'omnizart-' prefix to get the raw CLI mode name."""
|
||||
return mode_id.removeprefix("omnizart-")
|
||||
|
||||
|
||||
def _convert_to_wav(src: Path, dst: Path) -> None:
|
||||
"""
|
||||
Use ffmpeg to convert any audio file to WAV (mono, 44100 Hz).
|
||||
Also enforces MIN_DURATION_MS by adding silence padding if needed.
|
||||
"""
|
||||
cmd = [
|
||||
"ffmpeg", "-y",
|
||||
"-i", str(src),
|
||||
"-ar", "44100",
|
||||
"-ac", "1", # mono — Omnizart works best on mono
|
||||
"-af", f"apad=whole_dur={MIN_DURATION_MS}ms", # pad to min length
|
||||
str(dst),
|
||||
]
|
||||
subprocess.run(cmd, check=True, capture_output=True)
|
||||
|
||||
|
||||
def run_omnizart_transcription(job_id: str, input_path: str, mode_id: str) -> None:
|
||||
"""
|
||||
Background task: transcribe audio to MIDI using Omnizart CLI.
|
||||
mode_id: 'omnizart-music' | 'omnizart-vocal' | 'omnizart-vocal-contour'
|
||||
"""
|
||||
mode = _omnizart_mode(mode_id) # 'music', 'vocal', or 'vocal-contour'
|
||||
log.debug("[%s] omnizart start — mode=%s input=%s", job_id, mode, input_path)
|
||||
|
||||
output_dir = JOBS_DIR / job_id
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
jm.update_job(job_id, status=jm.JobStatus.RUNNING, progress="Converting to WAV...")
|
||||
|
||||
try:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
tmp_path = Path(tmp)
|
||||
stem = Path(input_path).stem
|
||||
wav_path = tmp_path / f"{stem}-{mode}.wav"
|
||||
mid_name = f"{stem}-{mode}.mid"
|
||||
|
||||
# --- WAV conversion + padding ---
|
||||
_convert_to_wav(Path(input_path), wav_path)
|
||||
|
||||
if jm.should_cancel(job_id):
|
||||
jm.update_job(job_id, status=jm.JobStatus.CANCELLED, progress="Cancelled.")
|
||||
return
|
||||
|
||||
jm.update_job(job_id, progress=f"Transcribing with Omnizart ({mode})...")
|
||||
|
||||
# --- Subprocess invocation ---
|
||||
# Vocal mode ignores -o; output lands in CWD as {stem}-vocal.mid
|
||||
if mode == "vocal":
|
||||
# Run from tmp dir so the auto-named file lands there
|
||||
cmd = f'{OMNIZART_BIN} vocal transcribe "{wav_path}"'
|
||||
expected_output = tmp_path / mid_name # {stem}-vocal.mid in tmp
|
||||
else:
|
||||
output_mid = tmp_path / mid_name
|
||||
cmd = f'{OMNIZART_BIN} {mode} transcribe -o "{output_mid}" "{wav_path}"'
|
||||
expected_output = output_mid
|
||||
|
||||
proc = subprocess.Popen(
|
||||
shlex.split(cmd),
|
||||
cwd=str(tmp_path), # Important for vocal mode CWD output
|
||||
shell=False,
|
||||
)
|
||||
|
||||
# Poll for cancellation every 5 s (mirrors Omnizart-UI pattern)
|
||||
import time
|
||||
while proc.poll() is None:
|
||||
if jm.should_cancel(job_id):
|
||||
proc.terminate()
|
||||
proc.wait()
|
||||
jm.update_job(job_id, status=jm.JobStatus.CANCELLED, progress="Cancelled.")
|
||||
return
|
||||
time.sleep(5)
|
||||
|
||||
if proc.returncode != 0:
|
||||
raise RuntimeError(f"omnizart exited with code {proc.returncode}")
|
||||
|
||||
# --- Copy output to jobs dir ---
|
||||
dest = output_dir / mid_name
|
||||
if not expected_output.exists():
|
||||
raise FileNotFoundError(
|
||||
f"Omnizart produced no output at {expected_output}. "
|
||||
"Input may be too short or unsupported."
|
||||
)
|
||||
dest.write_bytes(expected_output.read_bytes())
|
||||
|
||||
log.debug("[%s] omnizart complete — output=%s", job_id, dest.name)
|
||||
jm.update_job(
|
||||
job_id,
|
||||
status=jm.JobStatus.DONE,
|
||||
progress="Transcription complete.",
|
||||
files=[{"name": dest.name, "path": str(dest)}],
|
||||
)
|
||||
|
||||
except Exception as exc:
|
||||
log.error("[%s] omnizart failed: %s", job_id, exc)
|
||||
jm.update_job(job_id, status=jm.JobStatus.ERROR, error=str(exc), progress="Error.")
|
||||
```
|
||||
|
||||
**Key design choices vs. Omnizart-UI reference:**
|
||||
|
||||
| Concern | Omnizart-UI approach | StemEngine approach |
|
||||
|---|---|---|
|
||||
| WAV conversion | `SoundUtil` (custom pydub wrapper) | `ffmpeg` subprocess (already on PATH) |
|
||||
| Cancellation | DB-poll in while loop | `jm.should_cancel()` in while loop |
|
||||
| Vocal CWD quirk | Start process without `cwd`, track file separately | Set `cwd=tmp` so the auto-named `.mid` lands inside the temp dir |
|
||||
| Job state | Custom `JobController` | Existing `jm` (job_manager) |
|
||||
| Output storage | DB blob | Filesystem under `JOBS_DIR` (same as basic-pitch) |
|
||||
|
||||
---
|
||||
|
||||
### 3. `backend/services/transcriber_service.py`
|
||||
|
||||
Extend `run_transcription()` to branch on engine type:
|
||||
|
||||
```python
|
||||
"""
|
||||
MIDI transcription dispatcher.
|
||||
Routes to basic-pitch or Omnizart based on mode prefix.
|
||||
"""
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
from .. import job_manager as jm
|
||||
from ..config import JOBS_DIR, TRANSCRIBE_PRESETS, OMNIZART_ENABLED
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def run_transcription(job_id: str, input_path: str, mode: str):
|
||||
"""
|
||||
Background task: transcribe audio to MIDI.
|
||||
mode: 'polyphonic' | 'melody' → basic-pitch
|
||||
'omnizart-*' → Omnizart CLI
|
||||
"""
|
||||
if mode.startswith("omnizart-"):
|
||||
if not OMNIZART_ENABLED:
|
||||
jm.update_job(job_id, status=jm.JobStatus.ERROR,
|
||||
error="Omnizart is not enabled on this server.",
|
||||
progress="Error.")
|
||||
return
|
||||
from .omnizart_service import run_omnizart_transcription
|
||||
run_omnizart_transcription(job_id, input_path, mode)
|
||||
else:
|
||||
_run_basic_pitch(job_id, input_path, mode)
|
||||
|
||||
|
||||
def _run_basic_pitch(job_id: str, input_path: str, mode: str):
|
||||
"""Original basic-pitch path (unchanged)."""
|
||||
from basic_pitch.inference import predict
|
||||
from basic_pitch import ICASSP_2022_MODEL_PATH
|
||||
|
||||
log.debug("[%s] basic-pitch start — mode=%s", job_id, mode)
|
||||
output_dir = JOBS_DIR / job_id
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
jm.update_job(job_id, status=jm.JobStatus.RUNNING, progress="Loading basic-pitch model...")
|
||||
|
||||
try:
|
||||
if jm.should_cancel(job_id):
|
||||
jm.update_job(job_id, status=jm.JobStatus.CANCELLED, progress="Cancelled.")
|
||||
return
|
||||
|
||||
params = TRANSCRIBE_PRESETS.get(mode, TRANSCRIBE_PRESETS["polyphonic"])
|
||||
jm.update_job(job_id, progress=f"Transcribing with basic-pitch ({mode})...")
|
||||
|
||||
_, midi_data, _ = predict(input_path, ICASSP_2022_MODEL_PATH, **params)
|
||||
|
||||
if jm.should_cancel(job_id):
|
||||
jm.update_job(job_id, status=jm.JobStatus.CANCELLED, progress="Cancelled.")
|
||||
return
|
||||
|
||||
output_name = f"{Path(input_path).stem}-{mode}.mid"
|
||||
dest = output_dir / output_name
|
||||
midi_data.write(str(dest))
|
||||
|
||||
log.debug("[%s] basic-pitch complete — output=%s", job_id, dest.name)
|
||||
jm.update_job(
|
||||
job_id,
|
||||
status=jm.JobStatus.DONE,
|
||||
progress="Transcription complete.",
|
||||
files=[{"name": dest.name, "path": str(dest)}],
|
||||
)
|
||||
|
||||
except Exception as exc:
|
||||
log.debug("[%s] basic-pitch failed: %s", job_id, exc)
|
||||
jm.update_job(job_id, status=jm.JobStatus.ERROR, error=str(exc), progress="Error.")
|
||||
```
|
||||
|
||||
**No other callers change.** `project_reprocess_service.py` already calls
|
||||
`run_transcription(job_id, path, mode)` — it will automatically route correctly once
|
||||
the mode string is set correctly when the reprocess is triggered.
|
||||
|
||||
---
|
||||
|
||||
### 4. `backend/routers/transcribe.py`
|
||||
|
||||
**No structural changes required.** `VALID_MODES` is built from `TRANSCRIBE_MODES`
|
||||
at import time, so the new mode IDs are automatically accepted once config is updated:
|
||||
|
||||
```python
|
||||
VALID_MODES = {m["id"] for m in TRANSCRIBE_MODES} # already expands automatically
|
||||
```
|
||||
|
||||
**Optional hardening** — add a 503 guard for Omnizart-specific modes if you want the
|
||||
API to surface availability without relying on the service layer:
|
||||
|
||||
```python
|
||||
from ..config import OMNIZART_ENABLED
|
||||
|
||||
@router.post("")
|
||||
async def transcribe(file: UploadFile = File(...), mode: str = Form(DEFAULT_TRANSCRIBE_MODE)):
|
||||
if mode not in VALID_MODES:
|
||||
raise HTTPException(status_code=400, detail=f"Unknown mode: {mode}")
|
||||
if mode.startswith("omnizart-") and not OMNIZART_ENABLED:
|
||||
raise HTTPException(status_code=503, detail="Omnizart is not available on this server.")
|
||||
...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 5. `frontend/src/pages/ProjectDetail.tsx` (and standalone transcribe UI)
|
||||
|
||||
The frontend `GET /api/transcribe/modes` call already fetches the full mode list
|
||||
dynamically. The main changes needed:
|
||||
|
||||
1. **Group the picker by engine** — separate the mode list into two visual sections
|
||||
("Basic Pitch" and "Omnizart") using the `engine` field added to each mode
|
||||
entry in config.
|
||||
2. **Disable Omnizart options when unavailable** — add a `GET /api/transcribe/status`
|
||||
endpoint (or embed availability in the modes response) and grey out Omnizart
|
||||
entries when `OMNIZART_ENABLED=false`.
|
||||
3. **Hint text** — add a tooltip/description for each Omnizart mode explaining its
|
||||
strength (multi-instrument vs. melody/vocal).
|
||||
|
||||
Minimal change if full grouping is deferred: just ensure the new mode IDs appear
|
||||
in the `<select>` dropdown without visual changes — the backend already serves them.
|
||||
|
||||
---
|
||||
|
||||
### 6. `backend/routers/transcribe.py` — modes endpoint enhancement (optional)
|
||||
|
||||
Add an `available` flag to the modes list so the frontend can react without a
|
||||
second request:
|
||||
|
||||
```python
|
||||
@router.get("/modes")
|
||||
def get_modes():
|
||||
return [
|
||||
{**m, "available": True if m["engine"] == "basic-pitch" else OMNIZART_ENABLED}
|
||||
for m in TRANSCRIBE_MODES
|
||||
]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Environment / Deployment Strategy
|
||||
|
||||
Omnizart has heavy and conflicting Python dependencies (TensorFlow 2.x, madmom,
|
||||
specific numpy, etc.) that **cannot coexist** with the main StemEngine Python
|
||||
environment. Three strategies are viable:
|
||||
|
||||
### Option A — Sidecar Container (Recommended)
|
||||
|
||||
Run a minimal Omnizart container (similar to the existing `Omnizart-UI` Docker
|
||||
setup) as a sidecar service. StemEngine calls it over HTTP with the audio file as
|
||||
a multipart upload and receives the `.mid` back.
|
||||
|
||||
```
|
||||
stemengine <──HTTP──> omnizart-sidecar (port 8765)
|
||||
```
|
||||
|
||||
- Pros: Clean isolation, zero Python conflict, independently restartable.
|
||||
- Cons: Requires an HTTP client call in `omnizart_service.py` instead of
|
||||
subprocess; adds latency for file transfer.
|
||||
- Docker Compose addition: reference `docker/x/Omnizart-UI/docker-compose.yml`
|
||||
and wire into `stemengine.yml` with a shared network.
|
||||
|
||||
The `omnizart_service.py` draft above can be adapted for HTTP by replacing the
|
||||
subprocess block with an `httpx` request to the sidecar.
|
||||
|
||||
### Option B — conda run / venv activation prefix
|
||||
|
||||
If Omnizart is installed in a separate conda env or venv on the same host:
|
||||
|
||||
```python
|
||||
OMNIZART_BIN = "conda run -n omnizart omnizart"
|
||||
# or
|
||||
OMNIZART_BIN = "/opt/omnizart-venv/bin/omnizart"
|
||||
```
|
||||
|
||||
Controlled entirely by the `OMNIZART_BIN` env var already in the service design.
|
||||
|
||||
- Pros: Simpler than a sidecar, no network overhead.
|
||||
- Cons: `conda run` startup latency (~3-5 s per job), env management complexity.
|
||||
|
||||
### Option C — Same Container (Not Recommended)
|
||||
|
||||
Install Omnizart in the same container as StemEngine. Likely to cause dependency
|
||||
conflicts with PyTorch/basic-pitch versions.
|
||||
|
||||
---
|
||||
|
||||
**Recommended path for initial implementation:** Option B during development
|
||||
(quick to test), Option A for production deployment.
|
||||
|
||||
---
|
||||
|
||||
## Phased Delivery
|
||||
|
||||
### Phase 1 — `omnizart-music` mode (MVP)
|
||||
|
||||
- [ ] Add `omnizart-music` to `TRANSCRIBE_MODES` in `config.py`
|
||||
- [ ] Add `OMNIZART_ENABLED` env var to `config.py`
|
||||
- [ ] Create `backend/services/omnizart_service.py` with `music` mode only
|
||||
- [ ] Update `run_transcription()` dispatcher in `transcriber_service.py`
|
||||
- [ ] Add optional 503 guard in router
|
||||
- [ ] Set `OMNIZART_BIN` in `.env` / docker compose env
|
||||
- [ ] Smoke test: upload a 30 s piano stem, verify `.mid` download
|
||||
- [ ] Update `GET /api/transcribe/modes` to include `available` flag
|
||||
|
||||
### Phase 2 — `omnizart-vocal` and `omnizart-vocal-contour`
|
||||
|
||||
- [ ] Add both modes to `TRANSCRIBE_MODES`
|
||||
- [ ] Extend `omnizart_service.py` with vocal CWD quirk handling (already
|
||||
designed in the service draft above)
|
||||
- [ ] Test vocal mode with a separated vocals stem
|
||||
|
||||
### Phase 3 — Frontend polish & sidecar
|
||||
|
||||
- [ ] Group mode picker by engine in `ProjectDetail.tsx`
|
||||
- [ ] Grey out Omnizart modes when `available: false` from modes endpoint
|
||||
- [ ] Add tooltips/descriptions to mode entries
|
||||
- [ ] Migrate from subprocess / conda-run to sidecar HTTP call
|
||||
- [ ] Add Omnizart sidecar service to `stemengine.yml`
|
||||
|
||||
---
|
||||
|
||||
## Risks & Constraints
|
||||
|
||||
| Risk | Severity | Mitigation |
|
||||
|---|---|---|
|
||||
| Omnizart not on PATH / wrong env | High | `OMNIZART_ENABLED=false` default; 503 if selected |
|
||||
| Input shorter than 10 s | Medium | ffmpeg `apad` enforces minimum; clear error message |
|
||||
| `vocal` mode ignores `-o` flag | Medium | Handled by setting `cwd=tmp` so CWD output lands in temp dir |
|
||||
| Long processing time (5–20 min for long files) | Medium | Cancellation via process `terminate()` + `jm.should_cancel()` polling |
|
||||
| Python dep conflicts in same env | High | Option A/B isolation strategies above |
|
||||
| `chord`/`drums` Omnizart modes broken | Low | Not implemented; excluded from plan |
|
||||
| Output MIDI quality varies by input quality | Low | Documented in UI tooltips |
|
||||
|
||||
---
|
||||
|
||||
## Reference Files
|
||||
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `docker/x/Omnizart-UI/backend/src/transcriber.py` | Subprocess invocation, vocal quirk, cancellation pattern |
|
||||
| `docker/x/Omnizart-UI/backend/src/sound_util.py` | WAV conversion reference (replaced by ffmpeg here) |
|
||||
| `docker/x/StemEngine-0.3beta/backend/routers/transcribe.py` | `_STEM_PREFERENCE` ordering comment referencing Omnizart |
|
||||
| `backend/services/transcriber_service.py` | Current basic-pitch dispatch (to be extended) |
|
||||
| `backend/config.py` lines 195–215 | `TRANSCRIBE_MODES` and `TRANSCRIBE_PRESETS` |
|
||||
| `backend/routers/transcribe.py` | `VALID_MODES` auto-expansion pattern |
|
||||
+44
-3
@@ -191,13 +191,54 @@ OUTPUT_FORMATS = [
|
||||
]
|
||||
DEFAULT_OUTPUT_FORMAT = "mp3"
|
||||
|
||||
# basic-pitch transcription modes (mapped to inference parameter presets)
|
||||
# Transcription modes — basic-pitch (in-process) and Omnizart (CLI/sidecar)
|
||||
# Each entry carries an "engine" tag so the frontend can group/disable entries
|
||||
# and a human-readable "description" surfaced as tooltip text.
|
||||
TRANSCRIBE_MODES = [
|
||||
{"id": "polyphonic", "label": "Polyphonic"},
|
||||
{"id": "melody", "label": "Melody / Vocal"},
|
||||
{
|
||||
"id": "polyphonic",
|
||||
"label": "Polyphonic",
|
||||
"engine": "basic-pitch",
|
||||
"description": "General multi-instrument MIDI — basic-pitch model (fast, in-process)",
|
||||
},
|
||||
{
|
||||
"id": "melody",
|
||||
"label": "Melody / Vocal",
|
||||
"engine": "basic-pitch",
|
||||
"description": "Single-voice / vocal melody extraction — basic-pitch with melodia trick",
|
||||
},
|
||||
# ── Omnizart modes ──────────────────────────────────────────────────────
|
||||
# Require OMNIZART_ENABLED=true; served by the omnizart-sidecar container
|
||||
# (or a local Omnizart installation when OMNIZART_SIDECAR_URL is unset).
|
||||
{
|
||||
"id": "omnizart-music",
|
||||
"label": "Omnizart — Multi-instrument",
|
||||
"engine": "omnizart",
|
||||
"description": "Full-arrangement, instrument-aware MIDI — Omnizart 'music' mode (slow, GPU)",
|
||||
},
|
||||
{
|
||||
"id": "omnizart-vocal",
|
||||
"label": "Omnizart — Vocal Melody",
|
||||
"engine": "omnizart",
|
||||
"description": "Vocal melody extraction — Omnizart 'vocal' mode",
|
||||
},
|
||||
{
|
||||
"id": "omnizart-vocal-contour",
|
||||
"label": "Omnizart — Vocal Contour",
|
||||
"engine": "omnizart",
|
||||
"description": "Vocal pitch contour (F0-based) — Omnizart 'vocal-contour' mode",
|
||||
},
|
||||
]
|
||||
DEFAULT_TRANSCRIBE_MODE = "polyphonic"
|
||||
|
||||
# Omnizart availability — set OMNIZART_ENABLED=true to unlock the modes above.
|
||||
# When False the router returns 503 for any omnizart-* mode request.
|
||||
OMNIZART_ENABLED: bool = os.environ.get("OMNIZART_ENABLED", "false").lower() == "true"
|
||||
|
||||
# When set, omnizart_service will POST audio to this URL and retrieve the MIDI
|
||||
# (sidecar HTTP mode). When empty the service falls back to a local subprocess.
|
||||
OMNIZART_SIDECAR_URL: str = os.environ.get("OMNIZART_SIDECAR_URL", "").strip()
|
||||
|
||||
# basic-pitch inference parameter presets per mode
|
||||
TRANSCRIBE_PRESETS = {
|
||||
"polyphonic": {
|
||||
|
||||
@@ -5,7 +5,7 @@ from pathlib import Path
|
||||
from fastapi import APIRouter, File, Form, UploadFile, HTTPException
|
||||
|
||||
from .. import job_manager as jm
|
||||
from ..config import UPLOAD_DIR, PROJECTS_DIR, TRANSCRIBE_MODES, DEFAULT_TRANSCRIBE_MODE
|
||||
from ..config import UPLOAD_DIR, PROJECTS_DIR, TRANSCRIBE_MODES, DEFAULT_TRANSCRIBE_MODE, OMNIZART_ENABLED
|
||||
from ..services.transcriber_service import run_transcription
|
||||
|
||||
router = APIRouter(prefix="/api/transcribe", tags=["transcribe"])
|
||||
@@ -20,7 +20,18 @@ _AUDIO_EXTS = {".mp3", ".wav", ".flac"}
|
||||
|
||||
@router.get("/modes")
|
||||
def get_modes():
|
||||
return TRANSCRIBE_MODES
|
||||
"""Return all transcription modes with an 'available' flag.
|
||||
|
||||
Omnizart modes are marked available=False when OMNIZART_ENABLED is not set,
|
||||
so the frontend can grey them out without a second request.
|
||||
"""
|
||||
return [
|
||||
{
|
||||
**m,
|
||||
"available": True if m["engine"] == "basic-pitch" else OMNIZART_ENABLED,
|
||||
}
|
||||
for m in TRANSCRIBE_MODES
|
||||
]
|
||||
|
||||
|
||||
@router.post("")
|
||||
@@ -30,6 +41,8 @@ async def transcribe(
|
||||
):
|
||||
if mode not in VALID_MODES:
|
||||
raise HTTPException(status_code=400, detail=f"Unknown mode: {mode}")
|
||||
if mode.startswith("omnizart-") and not OMNIZART_ENABLED:
|
||||
raise HTTPException(status_code=503, detail="Omnizart is not available on this server (OMNIZART_ENABLED=false).")
|
||||
|
||||
job_id = jm.create_job("transcribe")
|
||||
upload_path = UPLOAD_DIR / job_id / file.filename
|
||||
@@ -53,6 +66,8 @@ async def transcribe_from_project(
|
||||
"""
|
||||
if mode not in VALID_MODES:
|
||||
raise HTTPException(status_code=400, detail=f"Unknown mode: {mode}")
|
||||
if mode.startswith("omnizart-") and not OMNIZART_ENABLED:
|
||||
raise HTTPException(status_code=503, detail="Omnizart is not available on this server (OMNIZART_ENABLED=false).")
|
||||
|
||||
stems_dir = PROJECTS_DIR / project_name / "Stems"
|
||||
if not stems_dir.exists():
|
||||
|
||||
@@ -0,0 +1,255 @@
|
||||
"""
|
||||
MIDI transcription via Omnizart.
|
||||
|
||||
Two execution paths are supported, controlled by env vars:
|
||||
|
||||
Sidecar HTTP (preferred / production):
|
||||
Set OMNIZART_SIDECAR_URL=http://omnizart-sidecar:9765
|
||||
The service POSTs the audio file to POST /transcribe on the sidecar and
|
||||
streams the returned .mid bytes to the jobs directory.
|
||||
|
||||
Local subprocess (development / Option B):
|
||||
Leave OMNIZART_SIDECAR_URL unset.
|
||||
The service invokes `omnizart {mode} transcribe ...` directly.
|
||||
Set OMNIZART_BIN to override the binary name/path (e.g. for conda envs).
|
||||
|
||||
Supported mode IDs (prefix "omnizart-" is stripped before CLI invocation):
|
||||
omnizart-music → omnizart music transcribe -o <out.mid> <in.wav>
|
||||
omnizart-vocal → omnizart vocal transcribe <in.wav>
|
||||
omnizart-vocal-contour → omnizart vocal-contour transcribe -o <out.mid> <in.wav>
|
||||
|
||||
Key Omnizart constraints baked into this service:
|
||||
• WAV input only — ffmpeg conversion is performed internally.
|
||||
• Minimum ~10 s — apad silence padding is added via ffmpeg to reach MIN_DURATION_MS.
|
||||
• vocal mode ignores -o — output lands in CWD as {stem}-vocal.mid;
|
||||
we set cwd=tmp so it is captured automatically.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import shlex
|
||||
import subprocess
|
||||
import tempfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from .. import job_manager as jm
|
||||
from ..config import JOBS_DIR, OMNIZART_SIDECAR_URL
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
# Override with OMNIZART_BIN for conda / venv setups, e.g.:
|
||||
# OMNIZART_BIN="conda run -n omnizart omnizart"
|
||||
OMNIZART_BIN: str = os.environ.get("OMNIZART_BIN", "omnizart")
|
||||
|
||||
# Omnizart silently fails on inputs shorter than ~10 s.
|
||||
MIN_DURATION_MS: int = 10_000
|
||||
|
||||
# Polling interval (seconds) for cancellation checks during subprocess execution.
|
||||
_POLL_INTERVAL: float = 5.0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Public entry point
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def run_omnizart_transcription(job_id: str, input_path: str, mode_id: str) -> None:
|
||||
"""
|
||||
Background task: transcribe *input_path* to MIDI using Omnizart.
|
||||
|
||||
mode_id: 'omnizart-music' | 'omnizart-vocal' | 'omnizart-vocal-contour'
|
||||
Routes to sidecar HTTP or local subprocess depending on OMNIZART_SIDECAR_URL.
|
||||
"""
|
||||
if OMNIZART_SIDECAR_URL:
|
||||
_run_via_sidecar(job_id, input_path, mode_id)
|
||||
else:
|
||||
_run_via_subprocess(job_id, input_path, mode_id)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Sidecar HTTP path (Phase 3 / production)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _run_via_sidecar(job_id: str, input_path: str, mode_id: str) -> None:
|
||||
"""
|
||||
POST the audio file to the omnizart-sidecar HTTP service and save the
|
||||
returned MIDI bytes to the jobs directory.
|
||||
|
||||
Sidecar API contract:
|
||||
POST /transcribe
|
||||
multipart form fields:
|
||||
file — audio file (any format; sidecar handles WAV conversion)
|
||||
mode — raw Omnizart mode (music | vocal | vocal-contour)
|
||||
200 response body — raw MIDI bytes (Content-Type: audio/midi)
|
||||
4xx / 5xx — JSON {"error": "<message>"}
|
||||
"""
|
||||
try:
|
||||
import httpx # type: ignore[import]
|
||||
except ImportError as exc:
|
||||
_fail(job_id, "httpx is required for sidecar mode: pip install httpx")
|
||||
return
|
||||
|
||||
mode = _strip_prefix(mode_id)
|
||||
stem = Path(input_path).stem
|
||||
mid_name = f"{stem}-{mode}.mid"
|
||||
output_dir = JOBS_DIR / job_id
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
log.debug("[%s] omnizart sidecar start — mode=%s url=%s", job_id, mode, OMNIZART_SIDECAR_URL)
|
||||
jm.update_job(job_id, status=jm.JobStatus.RUNNING, progress="Sending to Omnizart sidecar…")
|
||||
|
||||
try:
|
||||
url = OMNIZART_SIDECAR_URL.rstrip("/") + "/transcribe"
|
||||
|
||||
with open(input_path, "rb") as fh:
|
||||
with httpx.Client(timeout=1800) as client: # 30 min — Omnizart can be slow
|
||||
response = client.post(
|
||||
url,
|
||||
data={"mode": mode},
|
||||
files={"file": (Path(input_path).name, fh, "application/octet-stream")},
|
||||
)
|
||||
|
||||
if response.status_code != 200:
|
||||
_fail(job_id, f"Sidecar returned HTTP {response.status_code}: {response.text[:200]}")
|
||||
return
|
||||
|
||||
if jm.should_cancel(job_id):
|
||||
jm.update_job(job_id, status=jm.JobStatus.CANCELLED, progress="Cancelled.")
|
||||
return
|
||||
|
||||
dest = output_dir / mid_name
|
||||
dest.write_bytes(response.content)
|
||||
|
||||
log.debug("[%s] omnizart sidecar complete — output=%s", job_id, dest.name)
|
||||
jm.update_job(
|
||||
job_id,
|
||||
status=jm.JobStatus.DONE,
|
||||
progress="Transcription complete.",
|
||||
files=[{"name": dest.name, "path": str(dest)}],
|
||||
)
|
||||
|
||||
except Exception as exc:
|
||||
log.error("[%s] omnizart sidecar error: %s", job_id, exc)
|
||||
_fail(job_id, str(exc))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Local subprocess path (development / Option B)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _run_via_subprocess(job_id: str, input_path: str, mode_id: str) -> None:
|
||||
"""
|
||||
Convert the input to WAV, invoke `omnizart {mode} transcribe ...` as a
|
||||
subprocess, capture the output .mid file and store it in the jobs directory.
|
||||
"""
|
||||
mode = _strip_prefix(mode_id)
|
||||
stem = Path(input_path).stem
|
||||
mid_name = f"{stem}-{mode}.mid"
|
||||
output_dir = JOBS_DIR / job_id
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
log.debug("[%s] omnizart subprocess start — mode=%s input=%s", job_id, mode, input_path)
|
||||
jm.update_job(job_id, status=jm.JobStatus.RUNNING, progress="Converting to WAV…")
|
||||
|
||||
try:
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
tmp_path = Path(tmp)
|
||||
wav_path = tmp_path / f"{stem}-{mode}.wav"
|
||||
|
||||
# ── WAV conversion + minimum-length padding ──────────────────
|
||||
_convert_to_wav(Path(input_path), wav_path)
|
||||
|
||||
if jm.should_cancel(job_id):
|
||||
jm.update_job(job_id, status=jm.JobStatus.CANCELLED, progress="Cancelled.")
|
||||
return
|
||||
|
||||
jm.update_job(job_id, progress=f"Transcribing with Omnizart ({mode})…")
|
||||
|
||||
# ── Build subprocess command ──────────────────────────────────
|
||||
# vocal mode: -o is silently ignored; output lands in CWD as
|
||||
# {stem}-vocal.mid. We set cwd=tmp so it ends up inside tmp.
|
||||
if mode == "vocal":
|
||||
expected_output: Path = tmp_path / mid_name
|
||||
cmd = f'{OMNIZART_BIN} vocal transcribe "{wav_path}"'
|
||||
else:
|
||||
expected_output = tmp_path / mid_name
|
||||
cmd = f'{OMNIZART_BIN} {mode} transcribe -o "{expected_output}" "{wav_path}"'
|
||||
|
||||
log.debug("[%s] cmd=%s", job_id, cmd)
|
||||
|
||||
proc = subprocess.Popen(
|
||||
shlex.split(cmd),
|
||||
cwd=str(tmp_path), # keeps vocal-mode output inside tmp
|
||||
shell=False,
|
||||
)
|
||||
|
||||
# ── Poll for cancellation ─────────────────────────────────────
|
||||
while proc.poll() is None:
|
||||
if jm.should_cancel(job_id):
|
||||
proc.terminate()
|
||||
proc.wait()
|
||||
jm.update_job(job_id, status=jm.JobStatus.CANCELLED, progress="Cancelled.")
|
||||
return
|
||||
time.sleep(_POLL_INTERVAL)
|
||||
|
||||
if proc.returncode != 0:
|
||||
raise RuntimeError(f"omnizart exited with code {proc.returncode}")
|
||||
|
||||
# ── Copy output to persistent jobs dir ───────────────────────
|
||||
if not expected_output.exists():
|
||||
raise FileNotFoundError(
|
||||
f"Omnizart produced no output at {expected_output}. "
|
||||
"The input may be too short, or the Omnizart model failed."
|
||||
)
|
||||
|
||||
dest = output_dir / mid_name
|
||||
dest.write_bytes(expected_output.read_bytes())
|
||||
|
||||
log.debug("[%s] omnizart subprocess complete — output=%s", job_id, dest.name)
|
||||
jm.update_job(
|
||||
job_id,
|
||||
status=jm.JobStatus.DONE,
|
||||
progress="Transcription complete.",
|
||||
files=[{"name": dest.name, "path": str(dest)}],
|
||||
)
|
||||
|
||||
except Exception as exc:
|
||||
log.error("[%s] omnizart subprocess error: %s", job_id, exc)
|
||||
_fail(job_id, str(exc))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _strip_prefix(mode_id: str) -> str:
|
||||
"""'omnizart-music' → 'music', etc."""
|
||||
return mode_id.removeprefix("omnizart-")
|
||||
|
||||
|
||||
def _convert_to_wav(src: Path, dst: Path) -> None:
|
||||
"""
|
||||
Convert *src* to mono 44.1 kHz WAV, padding silence to MIN_DURATION_MS.
|
||||
Omnizart fails silently on inputs shorter than ~10 s.
|
||||
"""
|
||||
cmd = [
|
||||
"ffmpeg", "-y",
|
||||
"-i", str(src),
|
||||
"-ar", "44100",
|
||||
"-ac", "1", # mono
|
||||
"-af", f"apad=whole_dur={MIN_DURATION_MS}ms", # pad to minimum length
|
||||
str(dst),
|
||||
]
|
||||
result = subprocess.run(cmd, capture_output=True)
|
||||
if result.returncode != 0:
|
||||
raise RuntimeError(
|
||||
f"ffmpeg WAV conversion failed (code {result.returncode}):\n"
|
||||
+ result.stderr.decode(errors="replace")
|
||||
)
|
||||
|
||||
|
||||
def _fail(job_id: str, message: str) -> None:
|
||||
jm.update_job(job_id, status=jm.JobStatus.ERROR, error=message, progress="Error.")
|
||||
@@ -1,27 +1,52 @@
|
||||
"""
|
||||
MIDI transcription via basic-pitch (in-process Python API).
|
||||
Supports two modes mapped to inference parameter presets:
|
||||
polyphonic — general multi-instrument transcription
|
||||
melody — single-voice / vocal melody (melodia trick, min frequency)
|
||||
MIDI transcription dispatcher.
|
||||
|
||||
Routes to the appropriate engine based on the mode prefix:
|
||||
'polyphonic' | 'melody' → basic-pitch (in-process, fast)
|
||||
'omnizart-*' → Omnizart CLI or sidecar HTTP (requires OMNIZART_ENABLED=true)
|
||||
"""
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
from .. import job_manager as jm
|
||||
from ..config import JOBS_DIR, TRANSCRIBE_PRESETS
|
||||
from ..config import JOBS_DIR, TRANSCRIBE_PRESETS, OMNIZART_ENABLED
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def run_transcription(job_id: str, input_path: str, mode: str):
|
||||
def run_transcription(job_id: str, input_path: str, mode: str) -> None:
|
||||
"""
|
||||
Background task: transcribe audio to MIDI using basic-pitch.
|
||||
mode: 'polyphonic' | 'melody'
|
||||
Background task: transcribe audio to MIDI.
|
||||
|
||||
mode: 'polyphonic' | 'melody' → basic-pitch
|
||||
'omnizart-music' → Omnizart music mode
|
||||
'omnizart-vocal' → Omnizart vocal mode
|
||||
'omnizart-vocal-contour' → Omnizart vocal-contour mode
|
||||
"""
|
||||
if mode.startswith("omnizart-"):
|
||||
if not OMNIZART_ENABLED:
|
||||
jm.update_job(
|
||||
job_id,
|
||||
status=jm.JobStatus.ERROR,
|
||||
error="Omnizart is not enabled on this server (OMNIZART_ENABLED=false).",
|
||||
progress="Error.",
|
||||
)
|
||||
return
|
||||
from .omnizart_service import run_omnizart_transcription
|
||||
run_omnizart_transcription(job_id, input_path, mode)
|
||||
else:
|
||||
_run_basic_pitch(job_id, input_path, mode)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# basic-pitch (original engine — unchanged)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _run_basic_pitch(job_id: str, input_path: str, mode: str) -> None:
|
||||
from basic_pitch.inference import predict
|
||||
from basic_pitch import ICASSP_2022_MODEL_PATH
|
||||
|
||||
log.debug("[%s] run_transcription start — mode=%s input=%s", job_id, mode, input_path)
|
||||
log.debug("[%s] basic-pitch start — mode=%s input=%s", job_id, mode, input_path)
|
||||
output_dir = JOBS_DIR / job_id
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
@@ -49,7 +74,7 @@ def run_transcription(job_id: str, input_path: str, mode: str):
|
||||
dest = output_dir / output_name
|
||||
midi_data.write(str(dest))
|
||||
|
||||
log.debug("[%s] transcription complete — output=%s", job_id, dest.name)
|
||||
log.debug("[%s] basic-pitch complete — output=%s", job_id, dest.name)
|
||||
jm.update_job(
|
||||
job_id,
|
||||
status=jm.JobStatus.DONE,
|
||||
@@ -58,5 +83,5 @@ def run_transcription(job_id: str, input_path: str, mode: str):
|
||||
)
|
||||
|
||||
except Exception as exc:
|
||||
log.debug("[%s] transcription failed — %s: %s", job_id, type(exc).__name__, exc)
|
||||
log.debug("[%s] basic-pitch failed — %s: %s", job_id, type(exc).__name__, exc)
|
||||
jm.update_job(job_id, status=jm.JobStatus.ERROR, error=str(exc), progress="Error.")
|
||||
@@ -19,7 +19,7 @@
|
||||
<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent" />
|
||||
<meta name="apple-mobile-web-app-title" content="StemEngine" />
|
||||
<link rel="apple-touch-icon" href="/icon.svg" />
|
||||
<script type="module" crossorigin src="/assets/index-CHN4qgvq.js"></script>
|
||||
<script type="module" crossorigin src="/assets/index-Bp4Bk7Dp.js"></script>
|
||||
<link rel="stylesheet" crossorigin href="/assets/index-7HDOX98b.css">
|
||||
</head>
|
||||
<body>
|
||||
|
||||
@@ -7,8 +7,8 @@ import {
|
||||
ListMusic, Edit2, Check, X as XIcon, Wand2, RotateCcw, Copy, Users,
|
||||
Play, Pause, Square, Eye, EyeOff,
|
||||
} from 'lucide-react'
|
||||
import { getProjectDetail, transcribeProject, abcProject, mixProject, lrcProject, lyricTranscribeProject, downloadUrl, reprocessStem, renameStem, getPerInstrumentModels } from '../api/client'
|
||||
import type { ProjectDetail as PD, Job, JobFile } from '../types'
|
||||
import { getProjectDetail, transcribeProject, abcProject, mixProject, lrcProject, lyricTranscribeProject, downloadUrl, reprocessStem, renameStem, getPerInstrumentModels, getTranscribeModes } from '../api/client'
|
||||
import type { ProjectDetail as PD, Job, JobFile, ModeOption } from '../types'
|
||||
import DynamicMixPlayer from '../components/DynamicMixPlayer'
|
||||
import LyricsLivePreview from '../components/LyricsLivePreview'
|
||||
|
||||
@@ -20,11 +20,6 @@ function formatTime(s: number): string {
|
||||
return `${m}:${sec.toString().padStart(2, '0')}`
|
||||
}
|
||||
|
||||
const TRANSCRIBE_MODES = [
|
||||
{ id: 'polyphonic', label: 'Polyphonic' },
|
||||
{ id: 'melody', label: 'Melody / Vocal' },
|
||||
]
|
||||
|
||||
const STATIC_OPTIONS = [
|
||||
{ id: 'no_vocals', label: 'No Vocals' },
|
||||
{ id: 'no_drums', label: 'No Drums' },
|
||||
@@ -1072,15 +1067,66 @@ function MixesSection({
|
||||
|
||||
// ─── MIDI section ───────────────────────────────────────────────────────────
|
||||
|
||||
/** Stable fallback used while the mode list is loading. */
|
||||
const FALLBACK_MODES: ModeOption[] = [
|
||||
{ id: 'polyphonic', label: 'Polyphonic', engine: 'basic-pitch', available: true },
|
||||
{ id: 'melody', label: 'Melody/Vocal', engine: 'basic-pitch', available: true },
|
||||
]
|
||||
|
||||
/** Render a grouped <select> for transcription engine + mode. */
|
||||
function TranscribeModeSelect({
|
||||
value,
|
||||
onChange,
|
||||
modes,
|
||||
}: {
|
||||
value: string
|
||||
onChange: (id: string) => void
|
||||
modes: ModeOption[]
|
||||
}) {
|
||||
const basicPitch = modes.filter((m) => m.engine !== 'omnizart')
|
||||
const omnizart = modes.filter((m) => m.engine === 'omnizart')
|
||||
|
||||
const renderOption = (m: ModeOption) => (
|
||||
<option
|
||||
key={m.id}
|
||||
value={m.id}
|
||||
disabled={m.available === false}
|
||||
title={m.description ?? ''}
|
||||
>
|
||||
{m.label}{m.available === false ? ' (unavailable)' : ''}
|
||||
</option>
|
||||
)
|
||||
|
||||
return (
|
||||
<select
|
||||
value={value}
|
||||
onChange={(e) => onChange(e.target.value)}
|
||||
className="rounded-lg bg-white dark:bg-gray-900 border border-slate-300 dark:border-gray-700 px-2.5 py-1.5 text-xs text-slate-900 dark:text-gray-100 focus:outline-none focus:border-purple-400"
|
||||
title={modes.find((m) => m.id === value)?.description ?? ''}
|
||||
>
|
||||
{omnizart.length > 0 ? (
|
||||
<>
|
||||
<optgroup label="── Basic Pitch ──">{basicPitch.map(renderOption)}</optgroup>
|
||||
<optgroup label="── Omnizart ──">{omnizart.map(renderOption)}</optgroup>
|
||||
</>
|
||||
) : (
|
||||
basicPitch.map(renderOption)
|
||||
)}
|
||||
</select>
|
||||
)
|
||||
}
|
||||
|
||||
function StemMidiRow({
|
||||
projectId,
|
||||
stemName,
|
||||
transJob,
|
||||
modes,
|
||||
onAction,
|
||||
}: {
|
||||
projectId: string
|
||||
stemName: string
|
||||
transJob: Job | null
|
||||
modes: ModeOption[]
|
||||
onAction: () => void
|
||||
}) {
|
||||
const [mode, setMode] = useState('polyphonic')
|
||||
@@ -1107,15 +1153,7 @@ function StemMidiRow({
|
||||
<span className="text-xs font-medium text-slate-600 dark:text-gray-300 w-16 shrink-0 truncate capitalize">
|
||||
{label}
|
||||
</span>
|
||||
<select
|
||||
value={mode}
|
||||
onChange={(e) => setMode(e.target.value)}
|
||||
className="rounded-lg bg-white dark:bg-gray-900 border border-slate-300 dark:border-gray-700 px-2.5 py-1.5 text-xs text-slate-900 dark:text-gray-100 focus:outline-none focus:border-purple-400"
|
||||
>
|
||||
{TRANSCRIBE_MODES.map((m) => (
|
||||
<option key={m.id} value={m.id}>{m.label}</option>
|
||||
))}
|
||||
</select>
|
||||
<TranscribeModeSelect value={mode} onChange={setMode} modes={modes} />
|
||||
<button
|
||||
onClick={handleTranscribe}
|
||||
disabled={busy || !!isActive(transJob)}
|
||||
@@ -1154,6 +1192,19 @@ function MidiSection({
|
||||
const [bulkBusy, setBulkBusy] = useState(false)
|
||||
const [bulkErr, setBulkErr] = useState<string | null>(null)
|
||||
|
||||
// Fetch mode list from the backend (includes engine + available flags).
|
||||
// Falls back to the static list while loading so the UI is never blank.
|
||||
const { data: modes = FALLBACK_MODES } = useQuery({
|
||||
queryKey: ['transcribe-modes'],
|
||||
queryFn: getTranscribeModes,
|
||||
staleTime: Infinity,
|
||||
})
|
||||
|
||||
// Default bulk-transcribe uses omnizart-music when available, else polyphonic.
|
||||
const bulkMode = modes.find((m) => m.id === 'omnizart-music' && m.available !== false)
|
||||
? 'omnizart-music'
|
||||
: 'polyphonic'
|
||||
|
||||
const anyActive = stems.some((s) => isActive(transJobs[(s.stem ?? s.name)]))
|
||||
|
||||
async function handleTranscribeAll() {
|
||||
@@ -1162,7 +1213,7 @@ function MidiSection({
|
||||
await Promise.all(
|
||||
stems
|
||||
.filter((s) => !isActive(transJobs[(s.stem ?? s.name)]))
|
||||
.map((s) => transcribeProject(project.id, s.stem ?? s.name, 'music'))
|
||||
.map((s) => transcribeProject(project.id, s.stem ?? s.name, bulkMode))
|
||||
)
|
||||
onAction()
|
||||
} catch (e) {
|
||||
@@ -1213,6 +1264,7 @@ function MidiSection({
|
||||
projectId={project.id}
|
||||
stemName={stemName}
|
||||
transJob={transJobs[stemName] ?? null}
|
||||
modes={modes}
|
||||
onAction={onAction}
|
||||
/>
|
||||
)
|
||||
|
||||
@@ -45,6 +45,9 @@ export interface PerInstrumentModel {
|
||||
export interface ModeOption {
|
||||
id: string
|
||||
label: string
|
||||
engine?: string
|
||||
available?: boolean
|
||||
description?: string
|
||||
}
|
||||
|
||||
export interface FormatOption {
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
# Omnizart Sidecar
|
||||
#
|
||||
# Wraps the Omnizart CLI as an HTTP service consumed by the main StemEngine
|
||||
# backend. The base image must have Omnizart and its dependencies installed.
|
||||
#
|
||||
# Build:
|
||||
# docker build -t omnizart-sidecar omnizart-sidecar/
|
||||
#
|
||||
# The easiest base is the official Omnizart Docker image (if published), or
|
||||
# build from the Omnizart-UI backend image which already has Omnizart + deps.
|
||||
# Adjust FROM as needed for your Omnizart installation.
|
||||
|
||||
# ── Stage 1: use an image that already has Omnizart installed ───────────────
|
||||
# Replace this with the actual image that has omnizart on PATH.
|
||||
# Example options:
|
||||
# FROM music-ai-ntu/omnizart:latest (official, if available)
|
||||
# FROM your-registry/omnizart-base:latest
|
||||
# For development you can also build from the Omnizart-UI backend Dockerfile.
|
||||
FROM python:3.8-slim AS base
|
||||
|
||||
# Install system dependencies
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
ffmpeg \
|
||||
git \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install Omnizart and its Python dependencies
|
||||
# NOTE: Omnizart requires Python 3.8 and specific TensorFlow/numpy versions.
|
||||
# This step can take 10-20 min on first build due to TensorFlow.
|
||||
RUN pip install --no-cache-dir omnizart
|
||||
|
||||
# ── Stage 2: install sidecar app dependencies ────────────────────────────────
|
||||
RUN pip install --no-cache-dir fastapi uvicorn[standard] python-multipart
|
||||
|
||||
WORKDIR /app
|
||||
COPY app.py /app/app.py
|
||||
|
||||
EXPOSE 9765
|
||||
|
||||
# Download Omnizart models at build time to avoid runtime delays.
|
||||
# Comment out any models you don't need to reduce image size.
|
||||
RUN omnizart music download-model || true
|
||||
RUN omnizart vocal download-model || true
|
||||
|
||||
HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
|
||||
CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:9765/health')"
|
||||
|
||||
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "9765", "--workers", "1"]
|
||||
@@ -0,0 +1,137 @@
|
||||
"""
|
||||
Omnizart Sidecar — minimal FastAPI HTTP wrapper around the Omnizart CLI.
|
||||
|
||||
Single endpoint: POST /transcribe
|
||||
Request (multipart/form-data):
|
||||
file — audio file (any format; converted to WAV internally)
|
||||
mode — omnizart mode: music | vocal | vocal-contour
|
||||
Response 200: raw MIDI bytes (Content-Type: audio/midi)
|
||||
Response 4xx: JSON {"detail": "<message>"}
|
||||
|
||||
Environment variables:
|
||||
OMNIZART_BIN — binary name or full path (default: "omnizart")
|
||||
HOST — bind address (default: "0.0.0.0")
|
||||
PORT — bind port (default: "9765")
|
||||
|
||||
Run:
|
||||
uvicorn app:app --host 0.0.0.0 --port 9765
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import shlex
|
||||
import subprocess
|
||||
import tempfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi import FastAPI, File, Form, HTTPException, UploadFile
|
||||
from fastapi.responses import Response
|
||||
|
||||
log = logging.getLogger("omnizart-sidecar")
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
|
||||
|
||||
OMNIZART_BIN: str = os.environ.get("OMNIZART_BIN", "omnizart")
|
||||
MIN_DURATION_MS: int = 10_000
|
||||
|
||||
SUPPORTED_MODES = {"music", "vocal", "vocal-contour"}
|
||||
|
||||
app = FastAPI(title="Omnizart Sidecar", version="1.0")
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
def health():
|
||||
return {"status": "ok"}
|
||||
|
||||
|
||||
@app.post("/transcribe")
|
||||
async def transcribe(
|
||||
file: UploadFile = File(...),
|
||||
mode: str = Form("music"),
|
||||
) -> Response:
|
||||
if mode not in SUPPORTED_MODES:
|
||||
raise HTTPException(status_code=400, detail=f"Unsupported mode: {mode!r}. Choose from: {sorted(SUPPORTED_MODES)}")
|
||||
|
||||
audio_bytes = await file.read()
|
||||
stem = Path(file.filename or "audio").stem
|
||||
mid_name = f"{stem}-{mode}.mid"
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
tmp_path = Path(tmp)
|
||||
src_path = tmp_path / (file.filename or "audio")
|
||||
wav_path = tmp_path / f"{stem}-{mode}.wav"
|
||||
|
||||
# Write uploaded file
|
||||
src_path.write_bytes(audio_bytes)
|
||||
|
||||
# Convert to WAV with minimum-length padding
|
||||
_convert_to_wav(src_path, wav_path)
|
||||
|
||||
# Build Omnizart command
|
||||
if mode == "vocal":
|
||||
# vocal mode ignores -o; output lands in CWD as {stem}-vocal.mid
|
||||
expected_output = tmp_path / mid_name
|
||||
cmd = f'{OMNIZART_BIN} vocal transcribe "{wav_path}"'
|
||||
else:
|
||||
expected_output = tmp_path / mid_name
|
||||
cmd = f'{OMNIZART_BIN} {mode} transcribe -o "{expected_output}" "{wav_path}"'
|
||||
|
||||
log.info("running: %s", cmd)
|
||||
result = subprocess.run(
|
||||
shlex.split(cmd),
|
||||
cwd=str(tmp_path),
|
||||
capture_output=True,
|
||||
)
|
||||
|
||||
if result.returncode != 0:
|
||||
stderr = result.stderr.decode(errors="replace")
|
||||
log.error("omnizart failed (code %d): %s", result.returncode, stderr)
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail=f"Omnizart exited with code {result.returncode}. See sidecar logs.",
|
||||
)
|
||||
|
||||
if not expected_output.exists():
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail=(
|
||||
f"Omnizart produced no output at {expected_output.name}. "
|
||||
"Input may be too short or the model failed silently."
|
||||
),
|
||||
)
|
||||
|
||||
midi_bytes = expected_output.read_bytes()
|
||||
|
||||
log.info("transcription complete: %s (%d bytes)", mid_name, len(midi_bytes))
|
||||
return Response(
|
||||
content=midi_bytes,
|
||||
media_type="audio/midi",
|
||||
headers={"Content-Disposition": f'attachment; filename="{mid_name}"'},
|
||||
)
|
||||
|
||||
|
||||
def _convert_to_wav(src: Path, dst: Path) -> None:
|
||||
cmd = [
|
||||
"ffmpeg", "-y",
|
||||
"-i", str(src),
|
||||
"-ar", "44100",
|
||||
"-ac", "1",
|
||||
"-af", f"apad=whole_dur={MIN_DURATION_MS}ms",
|
||||
str(dst),
|
||||
]
|
||||
result = subprocess.run(cmd, capture_output=True)
|
||||
if result.returncode != 0:
|
||||
raise HTTPException(
|
||||
status_code=422,
|
||||
detail=(
|
||||
"ffmpeg WAV conversion failed. "
|
||||
+ result.stderr.decode(errors="replace")[:300]
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
uvicorn.run(app, host=os.environ.get("HOST", "0.0.0.0"), port=int(os.environ.get("PORT", "9765")))
|
||||
@@ -73,6 +73,12 @@ services:
|
||||
- SUBSONIC_USERNAME=${NAVUSER}
|
||||
- SUBSONIC_PASSWORD=${NAVPASS}
|
||||
- HF_TOKEN=${HF_TOKEN}
|
||||
# Omnizart MIDI transcription (optional — set OMNIZART_ENABLED=true to activate)
|
||||
# Set to 'true' once the omnizart-sidecar service is running.
|
||||
- OMNIZART_ENABLED=${OMNIZART_ENABLED:-false}
|
||||
# URL of the omnizart-sidecar container (used when OMNIZART_ENABLED=true).
|
||||
# Must be reachable from inside the stemengine container.
|
||||
- OMNIZART_SIDECAR_URL=${OMNIZART_SIDECAR_URL:-http://omnizart-sidecar:9765}
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
@@ -83,6 +89,43 @@ services:
|
||||
volumes:
|
||||
# Persists node_modules across container restarts; seeded from image on first start.
|
||||
stemengine-node-modules:
|
||||
|
||||
# Omnizart sidecar model cache — avoids re-downloading on restart.
|
||||
omnizart-model-cache:
|
||||
|
||||
# ── Omnizart Sidecar ──────────────────────────────────────────────────────
|
||||
# Build: docker compose -f stemengine.yml build omnizart-sidecar
|
||||
# Start: docker compose -f stemengine.yml up -d omnizart-sidecar
|
||||
# Then set OMNIZART_ENABLED=true and restart the stemengine service.
|
||||
omnizart-sidecar:
|
||||
container_name: omnizart-sidecar
|
||||
build:
|
||||
context: ${SEFOLDER}/omnizart-sidecar
|
||||
dockerfile: Dockerfile
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- t2_proxy
|
||||
ports:
|
||||
- "9765:9765" # expose for direct testing; remove in production
|
||||
volumes:
|
||||
# Cache Omnizart models between restarts (~500 MB)
|
||||
- omnizart-model-cache:/root/.omnizart
|
||||
environment:
|
||||
- PYTHONUNBUFFERED=1
|
||||
- OMNIZART_BIN=${OMNIZART_BIN:-omnizart}
|
||||
# No GPU reservation by default — Omnizart can run on CPU (slower).
|
||||
# Uncomment the block below to grant GPU access:
|
||||
# deploy:
|
||||
# resources:
|
||||
# reservations:
|
||||
# devices:
|
||||
# - driver: nvidia
|
||||
# count: 1
|
||||
# capabilities: [gpu]
|
||||
profiles:
|
||||
# Only started when you explicitly enable the omnizart profile:
|
||||
# docker compose -f stemengine.yml --profile omnizart up -d
|
||||
- omnizart
|
||||
networks:
|
||||
t2_proxy:
|
||||
external: true
|
||||
Reference in new issue
Block a user