feat: Omnizart MIDI transcription — phases 1-3

Phase 1 (omnizart-music MVP):
- config.py: add omnizart-music/vocal/vocal-contour to TRANSCRIBE_MODES with
  engine + description fields; add OMNIZART_ENABLED and OMNIZART_SIDECAR_URL
  env-var config constants
- transcriber_service.py: dispatch run_transcription() to basic-pitch or
  omnizart based on mode prefix; extract _run_basic_pitch() helper
- routers/transcribe.py: add 503 guard for omnizart modes when disabled;
  GET /modes now returns available flag per mode

Phase 2 (all three Omnizart modes):
- backend/services/omnizart_service.py (new): WAV conversion via ffmpeg with
  10 s minimum padding; subprocess invocation with cancellation polling;
  vocal mode CWD quirk handled by setting cwd=tmp; all three modes supported

Phase 3 (frontend polish + sidecar):
- types.ts: extend ModeOption with engine, available, description fields
- ProjectDetail.tsx: replace static TRANSCRIBE_MODES with live useQuery call
  to GET /api/transcribe/modes; add TranscribeModeSelect component with
  <optgroup> grouping by engine, disabled options for unavailable modes, and
  title= tooltips from description field; handleTranscribeAll auto-selects
  omnizart-music when available, falls back to polyphonic
- omnizart-sidecar/app.py + Dockerfile (new): minimal FastAPI HTTP wrapper
  around the Omnizart CLI; single POST /transcribe endpoint; GET /health
- stemengine.yml: add omnizart-sidecar service (profiles: [omnizart]); add
  OMNIZART_ENABLED and OMNIZART_SIDECAR_URL env vars to stemengine service
This commit is contained in:
translator committed 2026-03-14 13:00:36 +09:00
1 parent e5ca71f7c4
commit 5c2152c57c
11 files changed
+1152 -34

No files matched your search

+499
View File
@@ -0,0 +1,499 @@
# Omnizart MIDI Transcription Integration Plan
**Date:** 2026-03-14
**Status:** Planning
**Scope:** Add Omnizart as a second transcription engine alongside basic-pitch.
---
## Background
StemEngine currently offers MIDI transcription via **basic-pitch** (Spotify/Google,
in-process Python API) with two modes:
| ID | Label | Engine |
|----|-------|--------|
| `polyphonic` | Polyphonic | basic-pitch |
| `melody` | Melody / Vocal | basic-pitch |
**Omnizart** (Music AI NTU, 2021) is a separate toolkit for music transcription. It
operates via a CLI (`omnizart {mode} transcribe ...`) and produces standard `.mid`
output. A reference integration already exists in the workspace at
`docker/x/Omnizart-UI/` (Sanic-based standalone API). The reference implementation
was the primary source for this plan.
### Why add it?
- basic-pitch is excellent for melody extraction but struggles with polyphonic
arrangements at instrument level (piano-roll style).
- Omnizart's `music` mode does instrument-aware multi-track transcription — it is
the best freely available option for generating separated MIDI per instrument.
- The `vocal` mode produces a melody MIDI line comparable to basic-pitch's `melody`
mode but via a different model trained specifically on vocal contours.
- The `transcribe.py` router already has a `_STEM_PREFERENCE` list ordered for
Omnizart's `music` mode (pitched instruments first, drums excluded) — the
groundwork is already there.
---
## Omnizart Mode Inventory
| Omnizart mode | CLI arg | Output | Status |
|---|---|---|---|
| `music` | `omnizart music transcribe -o <out.mid> <in.wav>` | Multi-track MIDI | ✅ Works |
| `vocal` | `omnizart vocal transcribe <in.wav>` | Single-track melody MIDI | ✅ Works (quirky `-o` behaviour, see below) |
| `vocal-contour` | `omnizart vocal-contour transcribe -o <out.mid> <in.wav>` | Contour F0 MIDI | ✅ Works |
| `chord` | — | Chord symbols | ❌ Broken (numpy/Python conflict) |
| `drums` | — | Drum MIDI | ❌ Broken (Python 2.7 dep) |
**Vocal mode output path quirk:** The `-o` flag is silently ignored. The output file
is always written to the current working directory as
`./{input-stem}-vocal.mid`. This must be handled explicitly (see service design).
**Minimum file length:** Omnizart silently fails or produces empty output for inputs
shorter than ~10 seconds. The reference implementation enforces `10000 ms` as a
hard minimum via `SoundUtil.ConvertFileToWav(src, dst, 10_000)`. We must do the same.
---
## New Mode IDs for StemEngine
Three new mode IDs are proposed, prefixed with `omnizart-` to make the dispatch
branch unambiguous:
| New ID | Label | Omnizart CLI mode |
|---|---|---|
| `omnizart-music` | Omnizart — Multi-instrument | `music` |
| `omnizart-vocal` | Omnizart — Vocal Melody | `vocal` |
| `omnizart-vocal-contour` | Omnizart — Vocal Contour | `vocal-contour` |
Phase 1 ships `omnizart-music` only. Phases 2 and 3 add the remaining modes.
---
## Files to Change
### 1. `backend/config.py`
**Location:** line 195 (`TRANSCRIBE_MODES` list).
Extend the list with the new modes and add an availability flag so the frontend
can grey them out if Omnizart is not installed/reachable:
```python
# basic-pitch transcription modes
TRANSCRIBE_MODES = [
{"id": "polyphonic", "label": "Polyphonic", "engine": "basic-pitch"},
{"id": "melody", "label": "Melody / Vocal", "engine": "basic-pitch"},
# Omnizart modes — require Omnizart to be installed (see OMNIZART_ENABLED)
{"id": "omnizart-music", "label": "Omnizart — Multi-instrument", "engine": "omnizart"},
{"id": "omnizart-vocal", "label": "Omnizart — Vocal Melody", "engine": "omnizart"},
{"id": "omnizart-vocal-contour", "label": "Omnizart — Vocal Contour", "engine": "omnizart"},
]
DEFAULT_TRANSCRIBE_MODE = "polyphonic"
# Set via env var; when False the Omnizart modes are still listed but the
# router returns 503 if one is selected (allows UI to warn without hiding them).
OMNIZART_ENABLED = os.environ.get("OMNIZART_ENABLED", "false").lower() == "true"
```
No changes needed to `TRANSCRIBE_PRESETS` — those are basic-pitch only and
dispatched conditionally (see service below).
---
### 2. `backend/services/omnizart_service.py` *(new file)*
This service owns all Omnizart-specific logic: WAV conversion, minimum-length
padding, subprocess invocation, output path resolution, and cleanup.
```python
"""
MIDI transcription via Omnizart CLI (subprocess).
Omnizart must be available on PATH (or OMNIZART_BIN env var).
Supports modes: music | vocal | vocal-contour
"""
import logging
import os
import shlex
import subprocess
import tempfile
from pathlib import Path
from .. import job_manager as jm
from ..config import JOBS_DIR
log = logging.getLogger(__name__)
# CLI binary name / path — override with OMNIZART_BIN env var for venv/conda
OMNIZART_BIN = os.environ.get("OMNIZART_BIN", "omnizart")
# Omnizart fails silently on inputs shorter than ~10 s; pad/reject at this limit
MIN_DURATION_MS = 10_000
def _omnizart_mode(mode_id: str) -> str:
"""Strip the 'omnizart-' prefix to get the raw CLI mode name."""
return mode_id.removeprefix("omnizart-")
def _convert_to_wav(src: Path, dst: Path) -> None:
"""
Use ffmpeg to convert any audio file to WAV (mono, 44100 Hz).
Also enforces MIN_DURATION_MS by adding silence padding if needed.
"""
cmd = [
"ffmpeg", "-y",
"-i", str(src),
"-ar", "44100",
"-ac", "1", # mono — Omnizart works best on mono
"-af", f"apad=whole_dur={MIN_DURATION_MS}ms", # pad to min length
str(dst),
]
subprocess.run(cmd, check=True, capture_output=True)
def run_omnizart_transcription(job_id: str, input_path: str, mode_id: str) -> None:
"""
Background task: transcribe audio to MIDI using Omnizart CLI.
mode_id: 'omnizart-music' | 'omnizart-vocal' | 'omnizart-vocal-contour'
"""
mode = _omnizart_mode(mode_id) # 'music', 'vocal', or 'vocal-contour'
log.debug("[%s] omnizart start — mode=%s input=%s", job_id, mode, input_path)
output_dir = JOBS_DIR / job_id
output_dir.mkdir(parents=True, exist_ok=True)
jm.update_job(job_id, status=jm.JobStatus.RUNNING, progress="Converting to WAV...")
try:
with tempfile.TemporaryDirectory() as tmp:
tmp_path = Path(tmp)
stem = Path(input_path).stem
wav_path = tmp_path / f"{stem}-{mode}.wav"
mid_name = f"{stem}-{mode}.mid"
# --- WAV conversion + padding ---
_convert_to_wav(Path(input_path), wav_path)
if jm.should_cancel(job_id):
jm.update_job(job_id, status=jm.JobStatus.CANCELLED, progress="Cancelled.")
return
jm.update_job(job_id, progress=f"Transcribing with Omnizart ({mode})...")
# --- Subprocess invocation ---
# Vocal mode ignores -o; output lands in CWD as {stem}-vocal.mid
if mode == "vocal":
# Run from tmp dir so the auto-named file lands there
cmd = f'{OMNIZART_BIN} vocal transcribe "{wav_path}"'
expected_output = tmp_path / mid_name # {stem}-vocal.mid in tmp
else:
output_mid = tmp_path / mid_name
cmd = f'{OMNIZART_BIN} {mode} transcribe -o "{output_mid}" "{wav_path}"'
expected_output = output_mid
proc = subprocess.Popen(
shlex.split(cmd),
cwd=str(tmp_path), # Important for vocal mode CWD output
shell=False,
)
# Poll for cancellation every 5 s (mirrors Omnizart-UI pattern)
import time
while proc.poll() is None:
if jm.should_cancel(job_id):
proc.terminate()
proc.wait()
jm.update_job(job_id, status=jm.JobStatus.CANCELLED, progress="Cancelled.")
return
time.sleep(5)
if proc.returncode != 0:
raise RuntimeError(f"omnizart exited with code {proc.returncode}")
# --- Copy output to jobs dir ---
dest = output_dir / mid_name
if not expected_output.exists():
raise FileNotFoundError(
f"Omnizart produced no output at {expected_output}. "
"Input may be too short or unsupported."
)
dest.write_bytes(expected_output.read_bytes())
log.debug("[%s] omnizart complete — output=%s", job_id, dest.name)
jm.update_job(
job_id,
status=jm.JobStatus.DONE,
progress="Transcription complete.",
files=[{"name": dest.name, "path": str(dest)}],
)
except Exception as exc:
log.error("[%s] omnizart failed: %s", job_id, exc)
jm.update_job(job_id, status=jm.JobStatus.ERROR, error=str(exc), progress="Error.")
```
**Key design choices vs. Omnizart-UI reference:**
| Concern | Omnizart-UI approach | StemEngine approach |
|---|---|---|
| WAV conversion | `SoundUtil` (custom pydub wrapper) | `ffmpeg` subprocess (already on PATH) |
| Cancellation | DB-poll in while loop | `jm.should_cancel()` in while loop |
| Vocal CWD quirk | Start process without `cwd`, track file separately | Set `cwd=tmp` so the auto-named `.mid` lands inside the temp dir |
| Job state | Custom `JobController` | Existing `jm` (job_manager) |
| Output storage | DB blob | Filesystem under `JOBS_DIR` (same as basic-pitch) |
---
### 3. `backend/services/transcriber_service.py`
Extend `run_transcription()` to branch on engine type:
```python
"""
MIDI transcription dispatcher.
Routes to basic-pitch or Omnizart based on mode prefix.
"""
import logging
from pathlib import Path
from .. import job_manager as jm
from ..config import JOBS_DIR, TRANSCRIBE_PRESETS, OMNIZART_ENABLED
log = logging.getLogger(__name__)
def run_transcription(job_id: str, input_path: str, mode: str):
"""
Background task: transcribe audio to MIDI.
mode: 'polyphonic' | 'melody' → basic-pitch
'omnizart-*' → Omnizart CLI
"""
if mode.startswith("omnizart-"):
if not OMNIZART_ENABLED:
jm.update_job(job_id, status=jm.JobStatus.ERROR,
error="Omnizart is not enabled on this server.",
progress="Error.")
return
from .omnizart_service import run_omnizart_transcription
run_omnizart_transcription(job_id, input_path, mode)
else:
_run_basic_pitch(job_id, input_path, mode)
def _run_basic_pitch(job_id: str, input_path: str, mode: str):
"""Original basic-pitch path (unchanged)."""
from basic_pitch.inference import predict
from basic_pitch import ICASSP_2022_MODEL_PATH
log.debug("[%s] basic-pitch start — mode=%s", job_id, mode)
output_dir = JOBS_DIR / job_id
output_dir.mkdir(parents=True, exist_ok=True)
jm.update_job(job_id, status=jm.JobStatus.RUNNING, progress="Loading basic-pitch model...")
try:
if jm.should_cancel(job_id):
jm.update_job(job_id, status=jm.JobStatus.CANCELLED, progress="Cancelled.")
return
params = TRANSCRIBE_PRESETS.get(mode, TRANSCRIBE_PRESETS["polyphonic"])
jm.update_job(job_id, progress=f"Transcribing with basic-pitch ({mode})...")
_, midi_data, _ = predict(input_path, ICASSP_2022_MODEL_PATH, **params)
if jm.should_cancel(job_id):
jm.update_job(job_id, status=jm.JobStatus.CANCELLED, progress="Cancelled.")
return
output_name = f"{Path(input_path).stem}-{mode}.mid"
dest = output_dir / output_name
midi_data.write(str(dest))
log.debug("[%s] basic-pitch complete — output=%s", job_id, dest.name)
jm.update_job(
job_id,
status=jm.JobStatus.DONE,
progress="Transcription complete.",
files=[{"name": dest.name, "path": str(dest)}],
)
except Exception as exc:
log.debug("[%s] basic-pitch failed: %s", job_id, exc)
jm.update_job(job_id, status=jm.JobStatus.ERROR, error=str(exc), progress="Error.")
```
**No other callers change.** `project_reprocess_service.py` already calls
`run_transcription(job_id, path, mode)` — it will automatically route correctly once
the mode string is set correctly when the reprocess is triggered.
---
### 4. `backend/routers/transcribe.py`
**No structural changes required.** `VALID_MODES` is built from `TRANSCRIBE_MODES`
at import time, so the new mode IDs are automatically accepted once config is updated:
```python
VALID_MODES = {m["id"] for m in TRANSCRIBE_MODES} # already expands automatically
```
**Optional hardening** — add a 503 guard for Omnizart-specific modes if you want the
API to surface availability without relying on the service layer:
```python
from ..config import OMNIZART_ENABLED
@router.post("")
async def transcribe(file: UploadFile = File(...), mode: str = Form(DEFAULT_TRANSCRIBE_MODE)):
if mode not in VALID_MODES:
raise HTTPException(status_code=400, detail=f"Unknown mode: {mode}")
if mode.startswith("omnizart-") and not OMNIZART_ENABLED:
raise HTTPException(status_code=503, detail="Omnizart is not available on this server.")
...
```
---
### 5. `frontend/src/pages/ProjectDetail.tsx` (and standalone transcribe UI)
The frontend `GET /api/transcribe/modes` call already fetches the full mode list
dynamically. The main changes needed:
1. **Group the picker by engine** — separate the mode list into two visual sections
("Basic Pitch" and "Omnizart") using the `engine` field added to each mode
entry in config.
2. **Disable Omnizart options when unavailable** — add a `GET /api/transcribe/status`
endpoint (or embed availability in the modes response) and grey out Omnizart
entries when `OMNIZART_ENABLED=false`.
3. **Hint text** — add a tooltip/description for each Omnizart mode explaining its
strength (multi-instrument vs. melody/vocal).
Minimal change if full grouping is deferred: just ensure the new mode IDs appear
in the `<select>` dropdown without visual changes — the backend already serves them.
---
### 6. `backend/routers/transcribe.py` — modes endpoint enhancement (optional)
Add an `available` flag to the modes list so the frontend can react without a
second request:
```python
@router.get("/modes")
def get_modes():
return [
{**m, "available": True if m["engine"] == "basic-pitch" else OMNIZART_ENABLED}
for m in TRANSCRIBE_MODES
]
```
---
## Environment / Deployment Strategy
Omnizart has heavy and conflicting Python dependencies (TensorFlow 2.x, madmom,
specific numpy, etc.) that **cannot coexist** with the main StemEngine Python
environment. Three strategies are viable:
### Option A — Sidecar Container (Recommended)
Run a minimal Omnizart container (similar to the existing `Omnizart-UI` Docker
setup) as a sidecar service. StemEngine calls it over HTTP with the audio file as
a multipart upload and receives the `.mid` back.
```
stemengine <──HTTP──> omnizart-sidecar (port 8765)
```
- Pros: Clean isolation, zero Python conflict, independently restartable.
- Cons: Requires an HTTP client call in `omnizart_service.py` instead of
subprocess; adds latency for file transfer.
- Docker Compose addition: reference `docker/x/Omnizart-UI/docker-compose.yml`
and wire into `stemengine.yml` with a shared network.
The `omnizart_service.py` draft above can be adapted for HTTP by replacing the
subprocess block with an `httpx` request to the sidecar.
### Option B — conda run / venv activation prefix
If Omnizart is installed in a separate conda env or venv on the same host:
```python
OMNIZART_BIN = "conda run -n omnizart omnizart"
# or
OMNIZART_BIN = "/opt/omnizart-venv/bin/omnizart"
```
Controlled entirely by the `OMNIZART_BIN` env var already in the service design.
- Pros: Simpler than a sidecar, no network overhead.
- Cons: `conda run` startup latency (~3-5 s per job), env management complexity.
### Option C — Same Container (Not Recommended)
Install Omnizart in the same container as StemEngine. Likely to cause dependency
conflicts with PyTorch/basic-pitch versions.
---
**Recommended path for initial implementation:** Option B during development
(quick to test), Option A for production deployment.
---
## Phased Delivery
### Phase 1 — `omnizart-music` mode (MVP)
- [ ] Add `omnizart-music` to `TRANSCRIBE_MODES` in `config.py`
- [ ] Add `OMNIZART_ENABLED` env var to `config.py`
- [ ] Create `backend/services/omnizart_service.py` with `music` mode only
- [ ] Update `run_transcription()` dispatcher in `transcriber_service.py`
- [ ] Add optional 503 guard in router
- [ ] Set `OMNIZART_BIN` in `.env` / docker compose env
- [ ] Smoke test: upload a 30 s piano stem, verify `.mid` download
- [ ] Update `GET /api/transcribe/modes` to include `available` flag
### Phase 2 — `omnizart-vocal` and `omnizart-vocal-contour`
- [ ] Add both modes to `TRANSCRIBE_MODES`
- [ ] Extend `omnizart_service.py` with vocal CWD quirk handling (already
designed in the service draft above)
- [ ] Test vocal mode with a separated vocals stem
### Phase 3 — Frontend polish & sidecar
- [ ] Group mode picker by engine in `ProjectDetail.tsx`
- [ ] Grey out Omnizart modes when `available: false` from modes endpoint
- [ ] Add tooltips/descriptions to mode entries
- [ ] Migrate from subprocess / conda-run to sidecar HTTP call
- [ ] Add Omnizart sidecar service to `stemengine.yml`
---
## Risks & Constraints
| Risk | Severity | Mitigation |
|---|---|---|
| Omnizart not on PATH / wrong env | High | `OMNIZART_ENABLED=false` default; 503 if selected |
| Input shorter than 10 s | Medium | ffmpeg `apad` enforces minimum; clear error message |
| `vocal` mode ignores `-o` flag | Medium | Handled by setting `cwd=tmp` so CWD output lands in temp dir |
| Long processing time (5–20 min for long files) | Medium | Cancellation via process `terminate()` + `jm.should_cancel()` polling |
| Python dep conflicts in same env | High | Option A/B isolation strategies above |
| `chord`/`drums` Omnizart modes broken | Low | Not implemented; excluded from plan |
| Output MIDI quality varies by input quality | Low | Documented in UI tooltips |
---
## Reference Files
| File | Purpose |
|---|---|
| `docker/x/Omnizart-UI/backend/src/transcriber.py` | Subprocess invocation, vocal quirk, cancellation pattern |
| `docker/x/Omnizart-UI/backend/src/sound_util.py` | WAV conversion reference (replaced by ffmpeg here) |
| `docker/x/StemEngine-0.3beta/backend/routers/transcribe.py` | `_STEM_PREFERENCE` ordering comment referencing Omnizart |
| `backend/services/transcriber_service.py` | Current basic-pitch dispatch (to be extended) |
| `backend/config.py` lines 195–215 | `TRANSCRIBE_MODES` and `TRANSCRIBE_PRESETS` |
| `backend/routers/transcribe.py` | `VALID_MODES` auto-expansion pattern |
+44 -3
View File
@@ -191,13 +191,54 @@ OUTPUT_FORMATS = [
]
DEFAULT_OUTPUT_FORMAT = "mp3"
# basic-pitch transcription modes (mapped to inference parameter presets)
# Transcription modes — basic-pitch (in-process) and Omnizart (CLI/sidecar)
# Each entry carries an "engine" tag so the frontend can group/disable entries
# and a human-readable "description" surfaced as tooltip text.
TRANSCRIBE_MODES = [
{"id": "polyphonic", "label": "Polyphonic"},
{"id": "melody", "label": "Melody / Vocal"},
{
"id": "polyphonic",
"label": "Polyphonic",
"engine": "basic-pitch",
"description": "General multi-instrument MIDI — basic-pitch model (fast, in-process)",
},
{
"id": "melody",
"label": "Melody / Vocal",
"engine": "basic-pitch",
"description": "Single-voice / vocal melody extraction — basic-pitch with melodia trick",
},
# ── Omnizart modes ──────────────────────────────────────────────────────
# Require OMNIZART_ENABLED=true; served by the omnizart-sidecar container
# (or a local Omnizart installation when OMNIZART_SIDECAR_URL is unset).
{
"id": "omnizart-music",
"label": "Omnizart — Multi-instrument",
"engine": "omnizart",
"description": "Full-arrangement, instrument-aware MIDI — Omnizart 'music' mode (slow, GPU)",
},
{
"id": "omnizart-vocal",
"label": "Omnizart — Vocal Melody",
"engine": "omnizart",
"description": "Vocal melody extraction — Omnizart 'vocal' mode",
},
{
"id": "omnizart-vocal-contour",
"label": "Omnizart — Vocal Contour",
"engine": "omnizart",
"description": "Vocal pitch contour (F0-based) — Omnizart 'vocal-contour' mode",
},
]
DEFAULT_TRANSCRIBE_MODE = "polyphonic"
# Omnizart availability — set OMNIZART_ENABLED=true to unlock the modes above.
# When False the router returns 503 for any omnizart-* mode request.
OMNIZART_ENABLED: bool = os.environ.get("OMNIZART_ENABLED", "false").lower() == "true"
# When set, omnizart_service will POST audio to this URL and retrieve the MIDI
# (sidecar HTTP mode). When empty the service falls back to a local subprocess.
OMNIZART_SIDECAR_URL: str = os.environ.get("OMNIZART_SIDECAR_URL", "").strip()
# basic-pitch inference parameter presets per mode
TRANSCRIBE_PRESETS = {
"polyphonic": {
+17 -2
View File
@@ -5,7 +5,7 @@ from pathlib import Path
from fastapi import APIRouter, File, Form, UploadFile, HTTPException
from .. import job_manager as jm
from ..config import UPLOAD_DIR, PROJECTS_DIR, TRANSCRIBE_MODES, DEFAULT_TRANSCRIBE_MODE
from ..config import UPLOAD_DIR, PROJECTS_DIR, TRANSCRIBE_MODES, DEFAULT_TRANSCRIBE_MODE, OMNIZART_ENABLED
from ..services.transcriber_service import run_transcription
router = APIRouter(prefix="/api/transcribe", tags=["transcribe"])
@@ -20,7 +20,18 @@ _AUDIO_EXTS = {".mp3", ".wav", ".flac"}
@router.get("/modes")
def get_modes():
return TRANSCRIBE_MODES
"""Return all transcription modes with an 'available' flag.
Omnizart modes are marked available=False when OMNIZART_ENABLED is not set,
so the frontend can grey them out without a second request.
"""
return [
{
**m,
"available": True if m["engine"] == "basic-pitch" else OMNIZART_ENABLED,
}
for m in TRANSCRIBE_MODES
]
@router.post("")
@@ -30,6 +41,8 @@ async def transcribe(
):
if mode not in VALID_MODES:
raise HTTPException(status_code=400, detail=f"Unknown mode: {mode}")
if mode.startswith("omnizart-") and not OMNIZART_ENABLED:
raise HTTPException(status_code=503, detail="Omnizart is not available on this server (OMNIZART_ENABLED=false).")
job_id = jm.create_job("transcribe")
upload_path = UPLOAD_DIR / job_id / file.filename
@@ -53,6 +66,8 @@ async def transcribe_from_project(
"""
if mode not in VALID_MODES:
raise HTTPException(status_code=400, detail=f"Unknown mode: {mode}")
if mode.startswith("omnizart-") and not OMNIZART_ENABLED:
raise HTTPException(status_code=503, detail="Omnizart is not available on this server (OMNIZART_ENABLED=false).")
stems_dir = PROJECTS_DIR / project_name / "Stems"
if not stems_dir.exists():
+255
View File
@@ -0,0 +1,255 @@
"""
MIDI transcription via Omnizart.
Two execution paths are supported, controlled by env vars:
Sidecar HTTP (preferred / production):
Set OMNIZART_SIDECAR_URL=http://omnizart-sidecar:9765
The service POSTs the audio file to POST /transcribe on the sidecar and
streams the returned .mid bytes to the jobs directory.
Local subprocess (development / Option B):
Leave OMNIZART_SIDECAR_URL unset.
The service invokes `omnizart {mode} transcribe ...` directly.
Set OMNIZART_BIN to override the binary name/path (e.g. for conda envs).
Supported mode IDs (prefix "omnizart-" is stripped before CLI invocation):
omnizart-music → omnizart music transcribe -o <out.mid> <in.wav>
omnizart-vocal → omnizart vocal transcribe <in.wav>
omnizart-vocal-contour → omnizart vocal-contour transcribe -o <out.mid> <in.wav>
Key Omnizart constraints baked into this service:
• WAV input only — ffmpeg conversion is performed internally.
• Minimum ~10 s — apad silence padding is added via ffmpeg to reach MIN_DURATION_MS.
• vocal mode ignores -o — output lands in CWD as {stem}-vocal.mid;
we set cwd=tmp so it is captured automatically.
"""
from __future__ import annotations
import logging
import os
import shlex
import subprocess
import tempfile
import time
from pathlib import Path
from typing import Optional
from .. import job_manager as jm
from ..config import JOBS_DIR, OMNIZART_SIDECAR_URL
log = logging.getLogger(__name__)
# Override with OMNIZART_BIN for conda / venv setups, e.g.:
# OMNIZART_BIN="conda run -n omnizart omnizart"
OMNIZART_BIN: str = os.environ.get("OMNIZART_BIN", "omnizart")
# Omnizart silently fails on inputs shorter than ~10 s.
MIN_DURATION_MS: int = 10_000
# Polling interval (seconds) for cancellation checks during subprocess execution.
_POLL_INTERVAL: float = 5.0
# ---------------------------------------------------------------------------
# Public entry point
# ---------------------------------------------------------------------------
def run_omnizart_transcription(job_id: str, input_path: str, mode_id: str) -> None:
"""
Background task: transcribe *input_path* to MIDI using Omnizart.
mode_id: 'omnizart-music' | 'omnizart-vocal' | 'omnizart-vocal-contour'
Routes to sidecar HTTP or local subprocess depending on OMNIZART_SIDECAR_URL.
"""
if OMNIZART_SIDECAR_URL:
_run_via_sidecar(job_id, input_path, mode_id)
else:
_run_via_subprocess(job_id, input_path, mode_id)
# ---------------------------------------------------------------------------
# Sidecar HTTP path (Phase 3 / production)
# ---------------------------------------------------------------------------
def _run_via_sidecar(job_id: str, input_path: str, mode_id: str) -> None:
"""
POST the audio file to the omnizart-sidecar HTTP service and save the
returned MIDI bytes to the jobs directory.
Sidecar API contract:
POST /transcribe
multipart form fields:
file — audio file (any format; sidecar handles WAV conversion)
mode — raw Omnizart mode (music | vocal | vocal-contour)
200 response body — raw MIDI bytes (Content-Type: audio/midi)
4xx / 5xx — JSON {"error": "<message>"}
"""
try:
import httpx # type: ignore[import]
except ImportError as exc:
_fail(job_id, "httpx is required for sidecar mode: pip install httpx")
return
mode = _strip_prefix(mode_id)
stem = Path(input_path).stem
mid_name = f"{stem}-{mode}.mid"
output_dir = JOBS_DIR / job_id
output_dir.mkdir(parents=True, exist_ok=True)
log.debug("[%s] omnizart sidecar start — mode=%s url=%s", job_id, mode, OMNIZART_SIDECAR_URL)
jm.update_job(job_id, status=jm.JobStatus.RUNNING, progress="Sending to Omnizart sidecar…")
try:
url = OMNIZART_SIDECAR_URL.rstrip("/") + "/transcribe"
with open(input_path, "rb") as fh:
with httpx.Client(timeout=1800) as client: # 30 min — Omnizart can be slow
response = client.post(
url,
data={"mode": mode},
files={"file": (Path(input_path).name, fh, "application/octet-stream")},
)
if response.status_code != 200:
_fail(job_id, f"Sidecar returned HTTP {response.status_code}: {response.text[:200]}")
return
if jm.should_cancel(job_id):
jm.update_job(job_id, status=jm.JobStatus.CANCELLED, progress="Cancelled.")
return
dest = output_dir / mid_name
dest.write_bytes(response.content)
log.debug("[%s] omnizart sidecar complete — output=%s", job_id, dest.name)
jm.update_job(
job_id,
status=jm.JobStatus.DONE,
progress="Transcription complete.",
files=[{"name": dest.name, "path": str(dest)}],
)
except Exception as exc:
log.error("[%s] omnizart sidecar error: %s", job_id, exc)
_fail(job_id, str(exc))
# ---------------------------------------------------------------------------
# Local subprocess path (development / Option B)
# ---------------------------------------------------------------------------
def _run_via_subprocess(job_id: str, input_path: str, mode_id: str) -> None:
"""
Convert the input to WAV, invoke `omnizart {mode} transcribe ...` as a
subprocess, capture the output .mid file and store it in the jobs directory.
"""
mode = _strip_prefix(mode_id)
stem = Path(input_path).stem
mid_name = f"{stem}-{mode}.mid"
output_dir = JOBS_DIR / job_id
output_dir.mkdir(parents=True, exist_ok=True)
log.debug("[%s] omnizart subprocess start — mode=%s input=%s", job_id, mode, input_path)
jm.update_job(job_id, status=jm.JobStatus.RUNNING, progress="Converting to WAV…")
try:
with tempfile.TemporaryDirectory() as tmp:
tmp_path = Path(tmp)
wav_path = tmp_path / f"{stem}-{mode}.wav"
# ── WAV conversion + minimum-length padding ──────────────────
_convert_to_wav(Path(input_path), wav_path)
if jm.should_cancel(job_id):
jm.update_job(job_id, status=jm.JobStatus.CANCELLED, progress="Cancelled.")
return
jm.update_job(job_id, progress=f"Transcribing with Omnizart ({mode})…")
# ── Build subprocess command ──────────────────────────────────
# vocal mode: -o is silently ignored; output lands in CWD as
# {stem}-vocal.mid. We set cwd=tmp so it ends up inside tmp.
if mode == "vocal":
expected_output: Path = tmp_path / mid_name
cmd = f'{OMNIZART_BIN} vocal transcribe "{wav_path}"'
else:
expected_output = tmp_path / mid_name
cmd = f'{OMNIZART_BIN} {mode} transcribe -o "{expected_output}" "{wav_path}"'
log.debug("[%s] cmd=%s", job_id, cmd)
proc = subprocess.Popen(
shlex.split(cmd),
cwd=str(tmp_path), # keeps vocal-mode output inside tmp
shell=False,
)
# ── Poll for cancellation ─────────────────────────────────────
while proc.poll() is None:
if jm.should_cancel(job_id):
proc.terminate()
proc.wait()
jm.update_job(job_id, status=jm.JobStatus.CANCELLED, progress="Cancelled.")
return
time.sleep(_POLL_INTERVAL)
if proc.returncode != 0:
raise RuntimeError(f"omnizart exited with code {proc.returncode}")
# ── Copy output to persistent jobs dir ───────────────────────
if not expected_output.exists():
raise FileNotFoundError(
f"Omnizart produced no output at {expected_output}. "
"The input may be too short, or the Omnizart model failed."
)
dest = output_dir / mid_name
dest.write_bytes(expected_output.read_bytes())
log.debug("[%s] omnizart subprocess complete — output=%s", job_id, dest.name)
jm.update_job(
job_id,
status=jm.JobStatus.DONE,
progress="Transcription complete.",
files=[{"name": dest.name, "path": str(dest)}],
)
except Exception as exc:
log.error("[%s] omnizart subprocess error: %s", job_id, exc)
_fail(job_id, str(exc))
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _strip_prefix(mode_id: str) -> str:
"""'omnizart-music' → 'music', etc."""
return mode_id.removeprefix("omnizart-")
def _convert_to_wav(src: Path, dst: Path) -> None:
"""
Convert *src* to mono 44.1 kHz WAV, padding silence to MIN_DURATION_MS.
Omnizart fails silently on inputs shorter than ~10 s.
"""
cmd = [
"ffmpeg", "-y",
"-i", str(src),
"-ar", "44100",
"-ac", "1", # mono
"-af", f"apad=whole_dur={MIN_DURATION_MS}ms", # pad to minimum length
str(dst),
]
result = subprocess.run(cmd, capture_output=True)
if result.returncode != 0:
raise RuntimeError(
f"ffmpeg WAV conversion failed (code {result.returncode}):\n"
+ result.stderr.decode(errors="replace")
)
def _fail(job_id: str, message: str) -> None:
jm.update_job(job_id, status=jm.JobStatus.ERROR, error=message, progress="Error.")
+36 -11
View File
@@ -1,27 +1,52 @@
"""
MIDI transcription via basic-pitch (in-process Python API).
Supports two modes mapped to inference parameter presets:
polyphonic — general multi-instrument transcription
melody — single-voice / vocal melody (melodia trick, min frequency)
MIDI transcription dispatcher.
Routes to the appropriate engine based on the mode prefix:
'polyphonic' | 'melody' → basic-pitch (in-process, fast)
'omnizart-*' → Omnizart CLI or sidecar HTTP (requires OMNIZART_ENABLED=true)
"""
import logging
from pathlib import Path
from .. import job_manager as jm
from ..config import JOBS_DIR, TRANSCRIBE_PRESETS
from ..config import JOBS_DIR, TRANSCRIBE_PRESETS, OMNIZART_ENABLED
log = logging.getLogger(__name__)
def run_transcription(job_id: str, input_path: str, mode: str):
def run_transcription(job_id: str, input_path: str, mode: str) -> None:
"""
Background task: transcribe audio to MIDI using basic-pitch.
mode: 'polyphonic' | 'melody'
Background task: transcribe audio to MIDI.
mode: 'polyphonic' | 'melody' → basic-pitch
'omnizart-music' → Omnizart music mode
'omnizart-vocal' → Omnizart vocal mode
'omnizart-vocal-contour' → Omnizart vocal-contour mode
"""
if mode.startswith("omnizart-"):
if not OMNIZART_ENABLED:
jm.update_job(
job_id,
status=jm.JobStatus.ERROR,
error="Omnizart is not enabled on this server (OMNIZART_ENABLED=false).",
progress="Error.",
)
return
from .omnizart_service import run_omnizart_transcription
run_omnizart_transcription(job_id, input_path, mode)
else:
_run_basic_pitch(job_id, input_path, mode)
# ---------------------------------------------------------------------------
# basic-pitch (original engine — unchanged)
# ---------------------------------------------------------------------------
def _run_basic_pitch(job_id: str, input_path: str, mode: str) -> None:
from basic_pitch.inference import predict
from basic_pitch import ICASSP_2022_MODEL_PATH
log.debug("[%s] run_transcription start — mode=%s input=%s", job_id, mode, input_path)
log.debug("[%s] basic-pitch start — mode=%s input=%s", job_id, mode, input_path)
output_dir = JOBS_DIR / job_id
output_dir.mkdir(parents=True, exist_ok=True)
@@ -49,7 +74,7 @@ def run_transcription(job_id: str, input_path: str, mode: str):
dest = output_dir / output_name
midi_data.write(str(dest))
log.debug("[%s] transcription complete — output=%s", job_id, dest.name)
log.debug("[%s] basic-pitch complete — output=%s", job_id, dest.name)
jm.update_job(
job_id,
status=jm.JobStatus.DONE,
@@ -58,5 +83,5 @@ def run_transcription(job_id: str, input_path: str, mode: str):
)
except Exception as exc:
log.debug("[%s] transcription failed — %s: %s", job_id, type(exc).__name__, exc)
log.debug("[%s] basic-pitch failed — %s: %s", job_id, type(exc).__name__, exc)
jm.update_job(job_id, status=jm.JobStatus.ERROR, error=str(exc), progress="Error.")
+1 -1
View File
@@ -19,7 +19,7 @@
<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent" />
<meta name="apple-mobile-web-app-title" content="StemEngine" />
<link rel="apple-touch-icon" href="/icon.svg" />
<script type="module" crossorigin src="/assets/index-CHN4qgvq.js"></script>
<script type="module" crossorigin src="/assets/index-Bp4Bk7Dp.js"></script>
<link rel="stylesheet" crossorigin href="/assets/index-7HDOX98b.css">
</head>
<body>
+69 -17
View File
@@ -7,8 +7,8 @@ import {
ListMusic, Edit2, Check, X as XIcon, Wand2, RotateCcw, Copy, Users,
Play, Pause, Square, Eye, EyeOff,
} from 'lucide-react'
import { getProjectDetail, transcribeProject, abcProject, mixProject, lrcProject, lyricTranscribeProject, downloadUrl, reprocessStem, renameStem, getPerInstrumentModels } from '../api/client'
import type { ProjectDetail as PD, Job, JobFile } from '../types'
import { getProjectDetail, transcribeProject, abcProject, mixProject, lrcProject, lyricTranscribeProject, downloadUrl, reprocessStem, renameStem, getPerInstrumentModels, getTranscribeModes } from '../api/client'
import type { ProjectDetail as PD, Job, JobFile, ModeOption } from '../types'
import DynamicMixPlayer from '../components/DynamicMixPlayer'
import LyricsLivePreview from '../components/LyricsLivePreview'
@@ -20,11 +20,6 @@ function formatTime(s: number): string {
return `${m}:${sec.toString().padStart(2, '0')}`
}
const TRANSCRIBE_MODES = [
{ id: 'polyphonic', label: 'Polyphonic' },
{ id: 'melody', label: 'Melody / Vocal' },
]
const STATIC_OPTIONS = [
{ id: 'no_vocals', label: 'No Vocals' },
{ id: 'no_drums', label: 'No Drums' },
@@ -1072,15 +1067,66 @@ function MixesSection({
// ─── MIDI section ───────────────────────────────────────────────────────────
/** Stable fallback used while the mode list is loading. */
const FALLBACK_MODES: ModeOption[] = [
{ id: 'polyphonic', label: 'Polyphonic', engine: 'basic-pitch', available: true },
{ id: 'melody', label: 'Melody/Vocal', engine: 'basic-pitch', available: true },
]
/** Render a grouped <select> for transcription engine + mode. */
function TranscribeModeSelect({
value,
onChange,
modes,
}: {
value: string
onChange: (id: string) => void
modes: ModeOption[]
}) {
const basicPitch = modes.filter((m) => m.engine !== 'omnizart')
const omnizart = modes.filter((m) => m.engine === 'omnizart')
const renderOption = (m: ModeOption) => (
<option
key={m.id}
value={m.id}
disabled={m.available === false}
title={m.description ?? ''}
>
{m.label}{m.available === false ? ' (unavailable)' : ''}
</option>
)
return (
<select
value={value}
onChange={(e) => onChange(e.target.value)}
className="rounded-lg bg-white dark:bg-gray-900 border border-slate-300 dark:border-gray-700 px-2.5 py-1.5 text-xs text-slate-900 dark:text-gray-100 focus:outline-none focus:border-purple-400"
title={modes.find((m) => m.id === value)?.description ?? ''}
>
{omnizart.length > 0 ? (
<>
<optgroup label="── Basic Pitch ──">{basicPitch.map(renderOption)}</optgroup>
<optgroup label="── Omnizart ──">{omnizart.map(renderOption)}</optgroup>
</>
) : (
basicPitch.map(renderOption)
)}
</select>
)
}
function StemMidiRow({
projectId,
stemName,
transJob,
modes,
onAction,
}: {
projectId: string
stemName: string
transJob: Job | null
modes: ModeOption[]
onAction: () => void
}) {
const [mode, setMode] = useState('polyphonic')
@@ -1107,15 +1153,7 @@ function StemMidiRow({
<span className="text-xs font-medium text-slate-600 dark:text-gray-300 w-16 shrink-0 truncate capitalize">
{label}
</span>
<select
value={mode}
onChange={(e) => setMode(e.target.value)}
className="rounded-lg bg-white dark:bg-gray-900 border border-slate-300 dark:border-gray-700 px-2.5 py-1.5 text-xs text-slate-900 dark:text-gray-100 focus:outline-none focus:border-purple-400"
>
{TRANSCRIBE_MODES.map((m) => (
<option key={m.id} value={m.id}>{m.label}</option>
))}
</select>
<TranscribeModeSelect value={mode} onChange={setMode} modes={modes} />
<button
onClick={handleTranscribe}
disabled={busy || !!isActive(transJob)}
@@ -1154,6 +1192,19 @@ function MidiSection({
const [bulkBusy, setBulkBusy] = useState(false)
const [bulkErr, setBulkErr] = useState<string | null>(null)
// Fetch mode list from the backend (includes engine + available flags).
// Falls back to the static list while loading so the UI is never blank.
const { data: modes = FALLBACK_MODES } = useQuery({
queryKey: ['transcribe-modes'],
queryFn: getTranscribeModes,
staleTime: Infinity,
})
// Default bulk-transcribe uses omnizart-music when available, else polyphonic.
const bulkMode = modes.find((m) => m.id === 'omnizart-music' && m.available !== false)
? 'omnizart-music'
: 'polyphonic'
const anyActive = stems.some((s) => isActive(transJobs[(s.stem ?? s.name)]))
async function handleTranscribeAll() {
@@ -1162,7 +1213,7 @@ function MidiSection({
await Promise.all(
stems
.filter((s) => !isActive(transJobs[(s.stem ?? s.name)]))
.map((s) => transcribeProject(project.id, s.stem ?? s.name, 'music'))
.map((s) => transcribeProject(project.id, s.stem ?? s.name, bulkMode))
)
onAction()
} catch (e) {
@@ -1213,6 +1264,7 @@ function MidiSection({
projectId={project.id}
stemName={stemName}
transJob={transJobs[stemName] ?? null}
modes={modes}
onAction={onAction}
/>
)
+3
View File
@@ -45,6 +45,9 @@ export interface PerInstrumentModel {
export interface ModeOption {
id: string
label: string
engine?: string
available?: boolean
description?: string
}
export interface FormatOption {
+48
View File
@@ -0,0 +1,48 @@
# Omnizart Sidecar
#
# Wraps the Omnizart CLI as an HTTP service consumed by the main StemEngine
# backend. The base image must have Omnizart and its dependencies installed.
#
# Build:
# docker build -t omnizart-sidecar omnizart-sidecar/
#
# The easiest base is the official Omnizart Docker image (if published), or
# build from the Omnizart-UI backend image which already has Omnizart + deps.
# Adjust FROM as needed for your Omnizart installation.
# ── Stage 1: use an image that already has Omnizart installed ───────────────
# Replace this with the actual image that has omnizart on PATH.
# Example options:
# FROM music-ai-ntu/omnizart:latest (official, if available)
# FROM your-registry/omnizart-base:latest
# For development you can also build from the Omnizart-UI backend Dockerfile.
FROM python:3.8-slim AS base
# Install system dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
ffmpeg \
git \
&& rm -rf /var/lib/apt/lists/*
# Install Omnizart and its Python dependencies
# NOTE: Omnizart requires Python 3.8 and specific TensorFlow/numpy versions.
# This step can take 10-20 min on first build due to TensorFlow.
RUN pip install --no-cache-dir omnizart
# ── Stage 2: install sidecar app dependencies ────────────────────────────────
RUN pip install --no-cache-dir fastapi uvicorn[standard] python-multipart
WORKDIR /app
COPY app.py /app/app.py
EXPOSE 9765
# Download Omnizart models at build time to avoid runtime delays.
# Comment out any models you don't need to reduce image size.
RUN omnizart music download-model || true
RUN omnizart vocal download-model || true
HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:9765/health')"
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "9765", "--workers", "1"]
+137
View File
@@ -0,0 +1,137 @@
"""
Omnizart Sidecar — minimal FastAPI HTTP wrapper around the Omnizart CLI.
Single endpoint: POST /transcribe
Request (multipart/form-data):
file — audio file (any format; converted to WAV internally)
mode — omnizart mode: music | vocal | vocal-contour
Response 200: raw MIDI bytes (Content-Type: audio/midi)
Response 4xx: JSON {"detail": "<message>"}
Environment variables:
OMNIZART_BIN — binary name or full path (default: "omnizart")
HOST — bind address (default: "0.0.0.0")
PORT — bind port (default: "9765")
Run:
uvicorn app:app --host 0.0.0.0 --port 9765
"""
from __future__ import annotations
import logging
import os
import shlex
import subprocess
import tempfile
import time
from pathlib import Path
from fastapi import FastAPI, File, Form, HTTPException, UploadFile
from fastapi.responses import Response
log = logging.getLogger("omnizart-sidecar")
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
OMNIZART_BIN: str = os.environ.get("OMNIZART_BIN", "omnizart")
MIN_DURATION_MS: int = 10_000
SUPPORTED_MODES = {"music", "vocal", "vocal-contour"}
app = FastAPI(title="Omnizart Sidecar", version="1.0")
@app.get("/health")
def health():
return {"status": "ok"}
@app.post("/transcribe")
async def transcribe(
file: UploadFile = File(...),
mode: str = Form("music"),
) -> Response:
if mode not in SUPPORTED_MODES:
raise HTTPException(status_code=400, detail=f"Unsupported mode: {mode!r}. Choose from: {sorted(SUPPORTED_MODES)}")
audio_bytes = await file.read()
stem = Path(file.filename or "audio").stem
mid_name = f"{stem}-{mode}.mid"
with tempfile.TemporaryDirectory() as tmp:
tmp_path = Path(tmp)
src_path = tmp_path / (file.filename or "audio")
wav_path = tmp_path / f"{stem}-{mode}.wav"
# Write uploaded file
src_path.write_bytes(audio_bytes)
# Convert to WAV with minimum-length padding
_convert_to_wav(src_path, wav_path)
# Build Omnizart command
if mode == "vocal":
# vocal mode ignores -o; output lands in CWD as {stem}-vocal.mid
expected_output = tmp_path / mid_name
cmd = f'{OMNIZART_BIN} vocal transcribe "{wav_path}"'
else:
expected_output = tmp_path / mid_name
cmd = f'{OMNIZART_BIN} {mode} transcribe -o "{expected_output}" "{wav_path}"'
log.info("running: %s", cmd)
result = subprocess.run(
shlex.split(cmd),
cwd=str(tmp_path),
capture_output=True,
)
if result.returncode != 0:
stderr = result.stderr.decode(errors="replace")
log.error("omnizart failed (code %d): %s", result.returncode, stderr)
raise HTTPException(
status_code=500,
detail=f"Omnizart exited with code {result.returncode}. See sidecar logs.",
)
if not expected_output.exists():
raise HTTPException(
status_code=500,
detail=(
f"Omnizart produced no output at {expected_output.name}. "
"Input may be too short or the model failed silently."
),
)
midi_bytes = expected_output.read_bytes()
log.info("transcription complete: %s (%d bytes)", mid_name, len(midi_bytes))
return Response(
content=midi_bytes,
media_type="audio/midi",
headers={"Content-Disposition": f'attachment; filename="{mid_name}"'},
)
def _convert_to_wav(src: Path, dst: Path) -> None:
cmd = [
"ffmpeg", "-y",
"-i", str(src),
"-ar", "44100",
"-ac", "1",
"-af", f"apad=whole_dur={MIN_DURATION_MS}ms",
str(dst),
]
result = subprocess.run(cmd, capture_output=True)
if result.returncode != 0:
raise HTTPException(
status_code=422,
detail=(
"ffmpeg WAV conversion failed. "
+ result.stderr.decode(errors="replace")[:300]
),
)
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host=os.environ.get("HOST", "0.0.0.0"), port=int(os.environ.get("PORT", "9765")))
+43
View File
@@ -73,6 +73,12 @@ services:
- SUBSONIC_USERNAME=${NAVUSER}
- SUBSONIC_PASSWORD=${NAVPASS}
- HF_TOKEN=${HF_TOKEN}
# Omnizart MIDI transcription (optional — set OMNIZART_ENABLED=true to activate)
# Set to 'true' once the omnizart-sidecar service is running.
- OMNIZART_ENABLED=${OMNIZART_ENABLED:-false}
# URL of the omnizart-sidecar container (used when OMNIZART_ENABLED=true).
# Must be reachable from inside the stemengine container.
- OMNIZART_SIDECAR_URL=${OMNIZART_SIDECAR_URL:-http://omnizart-sidecar:9765}
deploy:
resources:
reservations:
@@ -83,6 +89,43 @@ services:
volumes:
# Persists node_modules across container restarts; seeded from image on first start.
stemengine-node-modules:
# Omnizart sidecar model cache — avoids re-downloading on restart.
omnizart-model-cache:
# ── Omnizart Sidecar ──────────────────────────────────────────────────────
# Build: docker compose -f stemengine.yml build omnizart-sidecar
# Start: docker compose -f stemengine.yml up -d omnizart-sidecar
# Then set OMNIZART_ENABLED=true and restart the stemengine service.
omnizart-sidecar:
container_name: omnizart-sidecar
build:
context: ${SEFOLDER}/omnizart-sidecar
dockerfile: Dockerfile
restart: unless-stopped
networks:
- t2_proxy
ports:
- "9765:9765" # expose for direct testing; remove in production
volumes:
# Cache Omnizart models between restarts (~500 MB)
- omnizart-model-cache:/root/.omnizart
environment:
- PYTHONUNBUFFERED=1
- OMNIZART_BIN=${OMNIZART_BIN:-omnizart}
# No GPU reservation by default — Omnizart can run on CPU (slower).
# Uncomment the block below to grant GPU access:
# deploy:
# resources:
# reservations:
# devices:
# - driver: nvidia
# count: 1
# capabilities: [gpu]
profiles:
# Only started when you explicitly enable the omnizart profile:
# docker compose -f stemengine.yml --profile omnizart up -d
- omnizart
networks:
t2_proxy:
external: true