Files
stemengine_old/20260316_UI_Models_Parakeet_Plan.md
T
2026-03-16 16:35:08 +09:00

6.8 KiB

StemEngine Rework Plan: Per-Instrument Separation + Parakeet Multilingual Transcription

Date: 2026-03-16

Goal

Rework StemEngine so users can configure per-instrument separation models and use the Parakeet multilingual transcription model, consistently across:

  • Main window (Separator flow)
  • Project pages
  • Auto-pipeline configuration and execution

Workstream 1 — Product Spec Lock

Status: ✅ Completed (2026-03-16)

Define and finalize:

  • Per-instrument mapping schema (e.g., vocals/drums/bass/other/guitar/piano -> model id)
  • Fallback behavior when a stem is missing
  • Parakeet scope (multilingual transcription; translation mode only if explicitly supported)
  • Output formats per engine (txt/lrc/elrc/json)
  • UI placement and defaults in main window, project pages, and auto-pipeline

Finalized contract decisions:

  • instrument_models is an optional JSON object passed by form field (multipart), e.g.:
    • {"vocals":"mel_band_vocals","drums":"mdxnet_drumsep"}
  • Allowed keys (canonical):
    • vocals, drums, bass, other, guitar, piano
  • Values must be valid existing model IDs from backend model registry.
  • Backward compatibility:
    • If instrument_models is absent, current behavior is unchanged.
    • Unknown keys/models are rejected with HTTP 400 + actionable message.
  • Fallback behavior:
    • Missing/invalid mapping does not block legacy separation requests.
    • Mapping is persisted in job/project metadata to support phased rollout.

Examples:

  • Separate endpoint form field:
    • instrument_models={"vocals":"mel_band_vocals","bass":"bs_roformer_4stem"}
  • Pipeline endpoint form field:
    • instrument_models={"vocals":"mdxnet_voc_ft"}

Workstream 2 — Backend API + Job Schema (Per-Instrument)

Status: ✅ Implemented (2026-03-16, updated)

Primary files:

  • backend/config.py
  • backend/routers/separate.py
  • backend/routers/pipeline.py
  • backend/services/separator_service.py
  • backend/services/project_reprocess_service.py

Plan:

  • Extend separation requests with instrument_models (JSON map)
  • Keep existing model as base/default for backward compatibility
  • Execute base separation + optional per-stem reprocess in one orchestrated flow
  • Persist chosen mapping in job result + project metadata
  • Ensure existing clients still work unchanged

Implemented now:

  • Added canonical instrument key constants in config.
  • Added shared parser/validator for instrument_models payloads.
  • Added instrument_models support to:
    • /api/separate
    • /api/separate/existing
    • /api/separate/from-path
    • /api/pipeline
  • Added optional in-job per-stem refinement execution in separation flow:
    • uses mapped model against each mapped base stem when available
    • promotes refined output as canonical stem path for that instrument
    • records applied/skipped details in job result + project.json
  • Persisted mapping summary into:
    • separation job result.instrument_models
    • project.json metadata (instrument_models)
    • pipeline job result.instrument_models.requested
  • Added pipeline lyric option pass-through plumbing:
    • /api/pipeline accepts optional lyric_engine and lyric_model
    • run_pipeline stores these values in job result.lyric
  • Preserved backward compatibility for all callers not sending the new field.

Current phased behavior note:

  • Mapping validation/persistence and separation-side refinement are enabled now.
  • Pipeline lyric engine/model pass-through is in place; execution behavior remains tied to later lyric-stage work.
  • Parakeet integration remains in later workstreams.

Workstream 3 — Parakeet as First-Class Engine

Status: 🟨 In progress (backend adapter + validation foundation implemented)

Primary files:

  • backend/services/lyric_transcriber_service.py
  • backend/routers/lyric_transcribe.py
  • backend/services/project_reprocess_service.py

Plan:

  • Add engine abstraction (whisperx, parakeet)
  • Use self-hosted Parakeet runtime only (no hosted trial/API coupling)
  • Validate engine + model in API layer
  • Expose available engines/models from lyric model endpoint
  • Enforce Parakeet multilingual language constraints server-side
  • Implement alternate handling for WhisperX-specific options (diarize, phoneme_align, lsync_align, two_pass)
  • Keep WhisperX defaults and behavior intact

Implemented now:

  • Added self-hosted Parakeet runtime settings in config (PARAKEET_*).
  • Added engine/model normalization helpers for WhisperX + Parakeet.
  • Added engine-aware lyric model metadata payload (engines + models by engine).
  • Added Parakeet runtime health check and transcribe adapter in lyric transcriber service.
  • Added Parakeet execution branch in run_lyric_transcription.
  • Added alternate handling for WhisperX-only options on Parakeet path (warnings + controlled ignore).
  • Added fail-fast validation for unsupported Parakeet translation task in router/service layers.
  • Updated lyric-transcribe endpoints to accept engine and validate model by engine.

Workstream 4 — Main Window UI (Separator)

Status: ✅ Completed (2026-03-16)

Primary files:

  • frontend/src/pages/Separator.tsx
  • frontend/src/api/client.ts
  • frontend/src/types.ts

Plan:

  • Add “Per-instrument refinement” panel with:
    • Enable toggle
    • Per-stem model selectors
    • Quick presets/profiles
  • Submit instrument_models with separation requests
  • Show selected mapping in result summary

Workstream 5 — Project Page UI

Status: ✅ Completed (2026-03-16)

Primary files:

  • frontend/src/pages/ProjectDetail.tsx
  • frontend/src/api/client.ts

Plan:

  • Add multi-stem/batch reprocess profile application
  • Add lyric engine selector (WhisperX vs Parakeet multilingual)
  • Filter models by selected engine
  • Save/reuse project-level defaults where practical

Workstream 6 — Auto-Pipeline

Status: ✅ Completed (2026-03-16)

Primary files:

  • frontend/src/utils/autoPipeline.ts
  • frontend/src/pages/Settings.tsx
  • frontend/src/components/AutoPipelineDialog.tsx
  • backend/routers/pipeline.py

Plan:

  • Extend step schema to include:
    • Per-instrument separation mapping
    • Lyric engine + model
  • Add migration for existing localStorage step objects
  • Ensure runner passes new fields end-to-end

Workstream 7 — Delivery Sequence

Status: 🟨 In progress (manual UI sign-off pending)

  1. Backend schema + compatibility layer
  2. Parakeet engine integration
  3. Main window UI updates
  4. Project page updates
  5. Auto-pipeline updates
  6. Tests, docs, rollout flags

Guardrails

  • Backward compatibility for existing API consumers and stored auto-pipeline steps
  • Feature-flagged rollout where possible
  • No regression in current WhisperX and standard separation paths

Remaining blocker

  • Backend/runtime Parakeet E2E is now passing for:
    • Project lyric-transcribe route
    • Direct upload lyric-transcribe route
  • Remaining sign-off is UI-level manual validation/documented approval for:
    • Project page control behavior
    • Auto-pipeline interaction flow