Files
pyannote-audio/tutorials/applying_a_model.ipynb
2022-11-29 18:14:49 +01:00

191 KiB

In [1]:
# preparing notebook for visualization purposes
# (only show outputs between t=0s and t=30s)
from pyannote.core import notebook, Segment
notebook.crop = Segment(0, 30)

Applying a pretrained model

In this tutorial, you will learn how to apply pyannote.audio models on an audio file, whose manual annotation is depicted below

In [2]:
# clone pyannote-audio Github repository and update ROOT_DIR accordingly
ROOT_DIR = "/Users/hbredin/Development/pyannote/pyannote-audio"
AUDIO_FILE = f"{ROOT_DIR}/tutorials/assets/sample.wav"
In [3]:
from pyannote.database.util import load_rttm
REFERENCE = f"{ROOT_DIR}/tutorials/assets/sample.rttm"
reference = load_rttm(REFERENCE)["sample"]
reference
Out [3]:

Loading models from 🤗 hub

A bunch of pretrained models are available on 🤗 Huggingface model hub and can be listed by looking for the pyannote-audio-model tag.

In [16]:
from huggingface_hub import HfApi
available_models = [m.modelId for m in HfApi().list_models(filter="pyannote-audio-model")]
list(filter(lambda p: p.startswith("pyannote/"), available_models))
Out [16]:
['pyannote/Segmentation-PyanNet-DIHARD',
 'pyannote/TestModelForContinuousIntegration',
 'pyannote/embedding',
 'pyannote/segmentation',
 'pyannote/brouhaha']

Official pyannote.audio models (i.e. those under the pyannote organization umbrella) are open-source, but gated. It means that you have to first accept users conditions on their respective Huggingface page to access the pretrained weights and hyper-parameters. Despite this initial process, those models can perfectly be downloaded for later offline use: keep reading this tutorial until the end to learn how to do that.

For instance, to load the speaker segmentation model used in this tutorial, you have to visit hf.co/pyannote/segmentation, accept the terms, and log in using notebook_login below:

In [5]:
from huggingface_hub import notebook_login
notebook_login()
VBox(children=(HTML(value='<center> <img\nsrc=https://huggingface.co/front/assets/huggingface_logo-noborder.sv…

Once authenticated, you can load the model...

In [6]:
from pyannote.audio import Model
model = Model.from_pretrained("pyannote/segmentation", use_auth_token=True)

... which consists in SincNet feature extraction (sincnet) , LSTM sequence modeling (lstm), a few feed-forward layers (linear), and a final multi-label classifier:

In [7]:
model.summarize()
Out [7]:
  | Name       | Type       | Params | In sizes      | Out sizes                                  
--------------------------------------------------------------------------------------------------------
0 | sincnet    | SincNet    | 42.6 K | [3, 1, 32000] | [3, 60, 115]                               
1 | lstm       | LSTM       | 1.4 M  | [3, 115, 60]  | [[3, 115, 256], [[8, 3, 128], [8, 3, 128]]]
2 | linear     | ModuleList | 49.4 K | ?             | ?                                          
3 | classifier | Linear     | 387    | [3, 115, 128] | [3, 115, 3]                                
4 | activation | Sigmoid    | 0      | [3, 115, 3]   | [3, 115, 3]                                
--------------------------------------------------------------------------------------------------------
1.5 M     Trainable params
0         Non-trainable params
1.5 M     Total params
5.891     Total estimated model params size (MB)

More details about the model are provided by its specifications...

In [8]:
specs = model.specifications
specs
Out [8]:
Specifications(problem=<Problem.MULTI_LABEL_CLASSIFICATION: 2>, resolution=<Resolution.FRAME: 1>, duration=5.0, warm_up=(0.0, 0.0), classes=['speaker#1', 'speaker#2', 'speaker#3'], permutation_invariant=True)

... which can be understood like that:

  • duration = 5.0: the model ingests 5s-long audio chunks
  • Resolution.FRAME and len(classes) == 3: the model output a sequence of frame-wise 3-dimensoinal scores
  • Problem.MULTI_LABEL_CLASSIFICATION for each frame, more than one speaker can be active at once

To apply the model on the audio file, we wrap it into an Inference instance:

In [9]:
from pyannote.audio import Inference
inference = Inference(model, step=2.5)
output = inference(AUDIO_FILE)
output
Out [9]:

For each of the 11 positions of the 5s window, the model outputs a 3-dimensional vector every 16ms (293 frames for 5 seconds), corresponding to the probabilities that each of (up to) 3 speakers is active.

In [10]:
output.data.shape
Out [10]:
(11, 293, 3)

Processing a file from memory

In case the audio file is not stored on disk, pipelines can also process audio provided as a {"waveform": ..., "sample_rate": ...} dictionary.

In [11]:
import torchaudio
waveform, sample_rate = torchaudio.load(AUDIO_FILE)

print(f"{type(waveform)=}")
print(f"{waveform.shape=}")
print(f"{waveform.dtype=}")

audio_in_memory = {"waveform": waveform, "sample_rate": sample_rate}
type(waveform)=<class 'torch.Tensor'>
waveform.shape=torch.Size([1, 480000])
waveform.dtype=torch.float32
In [12]:
output = inference(audio_in_memory)
output
Out [12]:

Processing part of a file

If needed, Inference can be used to process only part of a file:

In [13]:
from pyannote.core import Segment
output = inference.crop(AUDIO_FILE, Segment(10, 20))
output
Out [13]:

Offline use

Gating models allows me to know a bit more about pyannote.audio user base and eventually help me write grant proposals to make pyannote.audio even better. Please fill this form as precisely as possible.

For instance, before gating pyannote/segmentation, I had no idea that so many people were relying on it in production. Hint: sponsors are more than welcome! maintaining open source libraries is time consuming.

That being said: this whole authentication process does not prevent you from using official pyannote.audio models offline (i.e. without going through the authentication process in every docker run ... or whatever you are using in production).

  • Step 1: download the pytorch_model.bin model

  • Step 2: load the model
In [14]:
# look ma: no hands! 
offline_model = Model.from_pretrained("pytorch_model.bin")
In [15]:
# just checking weights are the same...
import torch
for weights, offline_weights in zip(model.parameters(), offline_model.parameters()):
    assert torch.equal(weights, offline_weights)