6.8 KiB
6.8 KiB
StemEngine Rework Plan: Per-Instrument Separation + Parakeet Multilingual Transcription
Date: 2026-03-16
Goal
Rework StemEngine so users can configure per-instrument separation models and use the Parakeet multilingual transcription model, consistently across:
- Main window (Separator flow)
- Project pages
- Auto-pipeline configuration and execution
Workstream 1 — Product Spec Lock
Status: ✅ Completed (2026-03-16)
Define and finalize:
- Per-instrument mapping schema (e.g., vocals/drums/bass/other/guitar/piano -> model id)
- Fallback behavior when a stem is missing
- Parakeet scope (multilingual transcription; translation mode only if explicitly supported)
- Output formats per engine (txt/lrc/elrc/json)
- UI placement and defaults in main window, project pages, and auto-pipeline
Finalized contract decisions:
instrument_modelsis an optional JSON object passed by form field (multipart), e.g.:{"vocals":"mel_band_vocals","drums":"mdxnet_drumsep"}
- Allowed keys (canonical):
vocals,drums,bass,other,guitar,piano
- Values must be valid existing model IDs from backend model registry.
- Backward compatibility:
- If
instrument_modelsis absent, current behavior is unchanged. - Unknown keys/models are rejected with HTTP 400 + actionable message.
- If
- Fallback behavior:
- Missing/invalid mapping does not block legacy separation requests.
- Mapping is persisted in job/project metadata to support phased rollout.
Examples:
- Separate endpoint form field:
instrument_models={"vocals":"mel_band_vocals","bass":"bs_roformer_4stem"}
- Pipeline endpoint form field:
instrument_models={"vocals":"mdxnet_voc_ft"}
Workstream 2 — Backend API + Job Schema (Per-Instrument)
Status: ✅ Implemented (2026-03-16, updated)
Primary files:
- backend/config.py
- backend/routers/separate.py
- backend/routers/pipeline.py
- backend/services/separator_service.py
- backend/services/project_reprocess_service.py
Plan:
- Extend separation requests with
instrument_models(JSON map) - Keep existing
modelas base/default for backward compatibility - Execute base separation + optional per-stem reprocess in one orchestrated flow
- Persist chosen mapping in job result + project metadata
- Ensure existing clients still work unchanged
Implemented now:
- Added canonical instrument key constants in config.
- Added shared parser/validator for
instrument_modelspayloads. - Added
instrument_modelssupport to:/api/separate/api/separate/existing/api/separate/from-path/api/pipeline
- Added optional in-job per-stem refinement execution in separation flow:
- uses mapped model against each mapped base stem when available
- promotes refined output as canonical stem path for that instrument
- records applied/skipped details in job result +
project.json
- Persisted mapping summary into:
- separation job
result.instrument_models project.jsonmetadata (instrument_models)- pipeline job
result.instrument_models.requested
- separation job
- Added pipeline lyric option pass-through plumbing:
/api/pipelineaccepts optionallyric_engineandlyric_modelrun_pipelinestores these values in jobresult.lyric
- Preserved backward compatibility for all callers not sending the new field.
Current phased behavior note:
- Mapping validation/persistence and separation-side refinement are enabled now.
- Pipeline lyric engine/model pass-through is in place; execution behavior remains tied to later lyric-stage work.
- Parakeet integration remains in later workstreams.
Workstream 3 — Parakeet as First-Class Engine
Status: 🟨 In progress (backend adapter + validation foundation implemented)
Primary files:
- backend/services/lyric_transcriber_service.py
- backend/routers/lyric_transcribe.py
- backend/services/project_reprocess_service.py
Plan:
- Add engine abstraction (
whisperx,parakeet) - Use self-hosted Parakeet runtime only (no hosted trial/API coupling)
- Validate
engine+modelin API layer - Expose available engines/models from lyric model endpoint
- Enforce Parakeet multilingual language constraints server-side
- Implement alternate handling for WhisperX-specific options (
diarize,phoneme_align,lsync_align,two_pass) - Keep WhisperX defaults and behavior intact
Implemented now:
- Added self-hosted Parakeet runtime settings in config (
PARAKEET_*). - Added engine/model normalization helpers for WhisperX + Parakeet.
- Added engine-aware lyric model metadata payload (engines + models by engine).
- Added Parakeet runtime health check and transcribe adapter in lyric transcriber service.
- Added Parakeet execution branch in
run_lyric_transcription. - Added alternate handling for WhisperX-only options on Parakeet path (warnings + controlled ignore).
- Added fail-fast validation for unsupported Parakeet translation task in router/service layers.
- Updated lyric-transcribe endpoints to accept
engineand validate model by engine.
Workstream 4 — Main Window UI (Separator)
Status: ✅ Completed (2026-03-16)
Primary files:
- frontend/src/pages/Separator.tsx
- frontend/src/api/client.ts
- frontend/src/types.ts
Plan:
- Add “Per-instrument refinement” panel with:
- Enable toggle
- Per-stem model selectors
- Quick presets/profiles
- Submit
instrument_modelswith separation requests - Show selected mapping in result summary
Workstream 5 — Project Page UI
Status: ✅ Completed (2026-03-16)
Primary files:
- frontend/src/pages/ProjectDetail.tsx
- frontend/src/api/client.ts
Plan:
- Add multi-stem/batch reprocess profile application
- Add lyric engine selector (WhisperX vs Parakeet multilingual)
- Filter models by selected engine
- Save/reuse project-level defaults where practical
Workstream 6 — Auto-Pipeline
Status: ✅ Completed (2026-03-16)
Primary files:
- frontend/src/utils/autoPipeline.ts
- frontend/src/pages/Settings.tsx
- frontend/src/components/AutoPipelineDialog.tsx
- backend/routers/pipeline.py
Plan:
- Extend step schema to include:
- Per-instrument separation mapping
- Lyric engine + model
- Add migration for existing localStorage step objects
- Ensure runner passes new fields end-to-end
Workstream 7 — Delivery Sequence
Status: 🟨 In progress (manual UI sign-off pending)
- Backend schema + compatibility layer
- Parakeet engine integration
- Main window UI updates
- Project page updates
- Auto-pipeline updates
- Tests, docs, rollout flags
Guardrails
- Backward compatibility for existing API consumers and stored auto-pipeline steps
- Feature-flagged rollout where possible
- No regression in current WhisperX and standard separation paths
Remaining blocker
- Backend/runtime Parakeet E2E is now passing for:
- Project lyric-transcribe route
- Direct upload lyric-transcribe route
- Remaining sign-off is UI-level manual validation/documented approval for:
- Project page control behavior
- Auto-pipeline interaction flow