Files

3.8 MiB

Open In Colab

Open In Colab

pyannote.audio is an open-source toolkit written in Python for speaker diarization.

Based on PyTorch machine learning framework, it provides a set of trainable end-to-end neural building blocks that can be combined and jointly optimized to build speaker diarization pipelines.

pyannote.audio also comes with pretrained models and pipelines covering a wide range of domains for voice activity detection, speaker segmentation, overlapped speech detection, speaker embedding reaching state-of-the-art performance for most of them.

This notebook will teach you how to apply those pretrained pipelines on your own data.

Make sure you run it using a GPU (or it might otherwise be slow...)

Installation

In [5]:
# for speechbrain
!pip install -qq torch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 torchtext==0.12.0
!pip install -qq speechbrain==0.5.12

# pyannote.audio
!pip install -qq pyannote.audio

# for visualization purposes
!pip install -qq moviepy ipython==7.34.0
     / 15.0 MB 349 kB/s
     |████████████████████████████████| 217 kB 25.8 MB/s 
     |████████████████████████████████| 79 kB 8.4 MB/s 
     |████████████████████████████████| 60 kB 6.9 MB/s 
     |████████████████████████████████| 41 kB 461 kB/s 
     |████████████████████████████████| 51 kB 165 kB/s 
     |████████████████████████████████| 585 kB 43.1 MB/s 
     |████████████████████████████████| 111 kB 63.2 MB/s 
     |████████████████████████████████| 47 kB 5.4 MB/s 
     |████████████████████████████████| 529 kB 64.4 MB/s 
     |████████████████████████████████| 117 kB 71.1 MB/s 
     |████████████████████████████████| 130 kB 67.2 MB/s 
     |████████████████████████████████| 348 kB 63.5 MB/s 
     |████████████████████████████████| 209 kB 70.7 MB/s 
     |████████████████████████████████| 81 kB 11.5 MB/s 
     |████████████████████████████████| 78 kB 8.1 MB/s 
     |████████████████████████████████| 59 kB 6.9 MB/s 
     |████████████████████████████████| 147 kB 60.6 MB/s 
     |████████████████████████████████| 112 kB 53.7 MB/s 
     |████████████████████████████████| 50 kB 6.0 MB/s 
[?25h  Building wheel for pyannote.audio (setup.py) ... [?25l[?25hdone
  Building wheel for antlr4-python3-runtime (setup.py) ... [?25l[?25hdone
  Building wheel for docopt (setup.py) ... [?25l[?25hdone
  Building wheel for julius (setup.py) ... [?25l[?25hdone
  Building wheel for pyperclip (setup.py) ... [?25l[?25hdone

Visualization with pyannote.core

For the purpose of this notebook, we will download and use an audio file coming from the AMI corpus, which contains a conversation between 4 people in a meeting room.

In [6]:
!wget -q http://groups.inf.ed.ac.uk/ami/AMICorpusMirror/amicorpus/ES2004a/audio/ES2004a.Mix-Headset.wav
DEMO_FILE = {'uri': 'ES2004a.Mix-Headset', 'audio': 'ES2004a.Mix-Headset.wav'}

Because AMI is a benchmarking dataset, it comes with manual annotations (a.k.a groundtruth).
Let us load and visualize the expected output of the speaker diarization pipeline.

In [7]:
!wget -q https://raw.githubusercontent.com/pyannote/AMI-diarization-setup/main/only_words/rttms/test/ES2004a.rttm
In [8]:
# load groundtruth
from pyannote.database.util import load_rttm
_, groundtruth = load_rttm('ES2004a.rttm').popitem()

# visualize groundtruth
groundtruth
Out [8]:

For the rest of this notebook, we will only listen to and visualize a one-minute long excerpt of the file (but will process the whole file anyway).

In [9]:
from pyannote.core import Segment, notebook
# make notebook visualization zoom on 600s < t < 660s time range
EXCERPT = Segment(600, 660)
notebook.crop = EXCERPT

# visualize excerpt groundtruth
groundtruth
Out [9]:

This nice visualization is brought to you by pyannote.core and basically indicates when each speaker speaks.

In [10]:
from pyannote.audio import Audio 
from IPython.display import Audio as IPythonAudio
waveform, sr = Audio().crop(DEMO_FILE, EXCERPT)
IPythonAudio(waveform.flatten(), rate=sr)
Out [10]:
Your browser does not support the audio element.

Processing your own audio file (optional)

In case you just want to go ahead with the demo file, skip this section entirely.

In case you want to try processing your own audio file, proceed with running this section. It will offer you to upload an audio file (preferably a wav file but all formats supported by SoundFile should work just fine).

Upload audio file

In [ ]:
import google.colab
own_file, _ = google.colab.files.upload().popitem()
OWN_FILE = {'audio': own_file}
notebook.reset()

# load audio waveform and play it
waveform, sample_rate = Audio()(OWN_FILE)
IPythonAudio(data=waveform.squeeze(), rate=sample_rate, autoplay=True)
Upload widget is only available when the cell has been executed in the current browser session. Please rerun this cell to enable.
Saving sample.wav to sample.wav
Your browser does not support the audio element.

Simply replace DEMO_FILE by OWN_FILE in the rest of the notebook.

Note, however, that unless you provide a groundtruth annotation in the next cell, you will (obviously) not be able to visualize groundtruth annotation nor evaluate the performance of the diarization pipeline quantitatively

Upload groundtruth (optional)

The groundtruth file is expected to use the RTTM format, with one line per speech turn with the following convention:

SPEAKER {file_name} 1 {start_time} {duration} <NA> <NA> {speaker_name} <NA> <NA>
In [ ]:
groundtruth_rttm, _ = google.colab.files.upload().popitem()
groundtruths = load_rttm(groundtruth_rttm)
if OWN_FILE['audio'] in groundtruths:
  groundtruth = groundtruths[OWN_FILE['audio']]
else:
  _, groundtruth = groundtruths.popitem()
groundtruth
Upload widget is only available when the cell has been executed in the current browser session. Please rerun this cell to enable.
Saving sample.rttm to sample.rttm

Speaker diarization with pyannote.pipeline

We are about to run a full speaker diarization pipeline, that includes speaker segmentation, speaker embedding, and a final clustering step. Brace yourself!

To load the speaker diarization pipeline,

In [11]:
from huggingface_hub import notebook_login
notebook_login()
Login successful
Your token has been saved to /root/.huggingface/token
In [12]:
from pyannote.audio import Pipeline
pipeline = Pipeline.from_pretrained('pyannote/speaker-diarization', use_auth_token=True)
diarization = pipeline(DEMO_FILE)
Downloading:   0%|          | 0.00/500 [00:00<?, ?B/s]
Downloading:   0%|          | 0.00/17.7M [00:00<?, ?B/s]
Downloading:   0%|          | 0.00/318 [00:00<?, ?B/s]
Downloading:   0%|          | 0.00/1.92k [00:00<?, ?B/s]
Downloading:   0%|          | 0.00/83.3M [00:00<?, ?B/s]
Downloading:   0%|          | 0.00/1.92k [00:00<?, ?B/s]
Downloading:   0%|          | 0.00/5.53M [00:00<?, ?B/s]
Downloading:   0%|          | 0.00/129k [00:00<?, ?B/s]

That's it? Yes, that's it :-)

In [13]:
diarization
Out [13]:

Evaluation with pyannote.metrics

Because groundtruth is available, we can evaluate the quality of the diarization pipeline by computing the diarization error rate.

In [14]:
from pyannote.metrics.diarization import DiarizationErrorRate
metric = DiarizationErrorRate()
der = metric(groundtruth, diarization)
In [15]:
print(f'diarization error rate = {100 * der:.1f}%')
diarization error rate = 40.2%

This implementation of diarization error rate is brought to you by pyannote.metrics.

It can also be used to improve visualization by find the optimal one-to-one mapping between groundtruth and hypothesized speakers.

In [16]:
mapping = metric.optimal_mapping(groundtruth, diarization)
diarization.rename_labels(mapping=mapping)
Out [16]:
In [17]:
groundtruth
Out [17]:

Going further

We have only scratched the surface in this introduction.

More details can be found in the pyannote.audio Github repository.