{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Offline Speaker Diarization (speaker-diarization-3.1)\n", "\n", "This notebooks gives a short introduction how to use the [speaker-diarization-3.1](https://huggingface.co/pyannote/speaker-diarization-3.1) pipeline with local models.\n", "\n", "In order to use local models, you first need to download them from huggingface and place them in a local folder. \n", "Then you need to create a local config file, similar to the one in HF, but with local model paths.\n", "\n", "❗ **Naming of the model files is REALLY important! See end of notebook for details.** ❗\n", "\n", "## Get the models\n", "\n", "1. Install the `pyannote-audio` package: `!pip install pyannote.audio`\n", "2. Create a huggingface account https://huggingface.co/join\n", "3. Accept [pyannote/segmentation-3.0](https://hf.co/pyannote/segmentation-3.0) user conditions\n", "4. Create a local folder `models`, place all downloaded files there\n", " 1. [wespeaker-voxceleb-resnet34-LM](https://huggingface.co/pyannote/wespeaker-voxceleb-resnet34-LM/blob/main/pytorch_model.bin), to be placed in `models/pyannote_model_wespeaker-voxceleb-resnet34-LM.bin`\n", " 2. [segmentation-3.0](https://huggingface.co/pyannote/segmentation-3.0/blob/main/pytorch_model.bin), to be placed in `models/pyannote_model_segmentation-3.0.bin`\n", "\n", "Running `ls models` should show the following files:\n", "```\n", "pyannote_model_segmentation-3.0.bin (5.7M)\n", "pyannote_model_wespeaker-voxceleb-resnet34-LM.bin (26MB)\n", "```\n", "\n", "❗ **make sure the 'wespeaker-voxceleb-resnet34-LM' model is named 'pyannote_model_wespeaker-voxceleb-resnet34-LM.bin'** ❗" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Config for local models\n", "\n", "Create a local config, similar to the one in HF: [speaker-diarization-3.1/blob/main/config.yaml](https://huggingface.co/pyannote/speaker-diarization-3.1/blob/main/config.yaml), but with local model paths\n", "\n", "Contents of `models/pyannote_diarization_config.yaml`:\n", "\n", "```yaml\n", "version: 3.1.0\n", "\n", "pipeline:\n", " name: pyannote.audio.pipelines.SpeakerDiarization\n", " params:\n", " clustering: AgglomerativeClustering\n", " # embedding: pyannote/wespeaker-voxceleb-resnet34-LM # if you want to use the HF model\n", " embedding: models/pyannote_model_wespeaker-voxceleb-resnet34-LM.bin # if you want to use the local model\n", " embedding_batch_size: 32\n", " embedding_exclude_overlap: true\n", " # segmentation: pyannote/segmentation-3.0 # if you want to use the HF model\n", " segmentation: models/pyannote_model_segmentation-3.0.bin # if you want to use the local model\n", " segmentation_batch_size: 32\n", "\n", "params:\n", " clustering:\n", " method: centroid\n", " min_cluster_size: 12\n", " threshold: 0.7045654963945799\n", " segmentation:\n", " min_duration_off: 0.0\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Loading the local pipeline\n", "\n", "**Hint**: The paths in the config are relative to the current working directory, not relative to the config file.\n", "If you want to start your notebook/script from a different directory, you can use `os.chdir` temporarily, to 'emulate' config-relative paths.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "from pyannote.audio import Pipeline\n", "\n", "def load_pipeline_from_pretrained(path_to_config: str | Path) -> Pipeline:\n", " path_to_config = Path(path_to_config)\n", "\n", " print(f\"Loading pyannote pipeline from {path_to_config}...\")\n", " # the paths in the config are relative to the current working directory\n", " # so we need to change the working directory to the model path\n", " # and then change it back\n", "\n", " cwd = Path.cwd().resolve() # store current working directory\n", "\n", " # first .parent is the folder of the config, second .parent is the folder containing the 'models' folder\n", " cd_to = path_to_config.parent.parent.resolve()\n", "\n", " print(f\"Changing working directory to {cd_to}\")\n", " os.chdir(cd_to)\n", "\n", " pipeline = Pipeline.from_pretrained(path_to_config)\n", "\n", " print(f\"Changing working directory back to {cwd}\")\n", " os.chdir(cwd)\n", "\n", " return pipeline\n", "\n", "PATH_TO_CONFIG = \"path/to/your/pyannote_diarization_config.yaml\"\n", "pipeline = load_pipeline_from_pretrained(PATH_TO_CONFIG)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Notes on file naming (pyannote-audio 3.1.1)\n", "\n", "Pyannote uses some internal logic to determine the model type.\n", "\n", "The function `def PretrainedSpeakerEmbedding(...` in (speaker_verification.py)[https://github.com/pyannote/pyannote-audio/blob/develop/pyannote/audio/pipelines/speaker_verification.py#L712] uses the the file path of the model to infer the model type.\n", "\n", "```python\n", "def PretrainedSpeakerEmbedding(\n", " embedding: PipelineModel,\n", " device: torch.device = None,\n", " use_auth_token: Union[Text, None] = None,\n", "):\n", " #...\n", " if isinstance(embedding, str) and \"pyannote\" in embedding:\n", " return PyannoteAudioPretrainedSpeakerEmbedding(\n", " embedding, device=device, use_auth_token=use_auth_token\n", " )\n", "\n", " elif isinstance(embedding, str) and \"speechbrain\" in embedding:\n", " return SpeechBrainPretrainedSpeakerEmbedding(\n", " embedding, device=device, use_auth_token=use_auth_token\n", " )\n", "\n", " elif isinstance(embedding, str) and \"nvidia\" in embedding:\n", " return NeMoPretrainedSpeakerEmbedding(embedding, device=device)\n", "\n", " elif isinstance(embedding, str) and \"wespeaker\" in embedding:\n", " return ONNXWeSpeakerPretrainedSpeakerEmbedding(embedding, device=device) # <-- this is called, but the wespeaker-voxceleb-resnet34-LM is not an ONNX model\n", "\n", " else:\n", " # fallback to pyannote in case we are loading a local model\n", " return PyannoteAudioPretrainedSpeakerEmbedding(\n", " embedding, device=device, use_auth_token=use_auth_token\n", " )\n", "```\n", "\n", "The [wespeaker-voxceleb-resnet34-LM](https://huggingface.co/pyannote/wespeaker-voxceleb-resnet34-LM/blob/main/pytorch_model.bin) model is not an ONNX model, but a `PyannoteAudioPretrainedSpeakerEmbedding`. So if `wespeaker` is in the file name, the code will infer the model type incorrectly. If `pyannote` is somewhere in the file name, the model type will be inferred correctly, as the first if statement will be true..." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.11.7" } }, "nbformat": 4, "nbformat_minor": 2 }