Files
pyannote-audio/notebook/example.ipynb
T
Hervé BREDIN 7121a73f08 feat: add support for label scope and missing classes (#1296)
- setup: switch to pyannote.database 5.0
- feat(task): add support for label scope in speaker diarization task (from pyannote.database 5.0)
- feat(task): add support for missing classes in multi-label segmentation task (from pyannote.database 5.0)
- improve(task): load metadata as tensors/arrays rather than pyannote.core instances
- BREAKING(task): rename `Segmentation` task to `SpeakerDiarization`
- BREAKING(task): remove support for variable chunk duration
2023-04-13 21:15:57 +02:00

1.2 MiB

In [1]:
from pyannote.database import registry, FileFinder
registry.load_database('../tests/data/database.yml')
protocol = registry.get_protocol('Debug.SpeakerDiarization.Debug', 
                                 preprocessors={"audio": FileFinder()})
'Debug.SpeakerDiarization.Debug' found in /Users/hbredin/Development/pyannote/pyannote-audio/tests/data/database.yml does not define the 'scope' of speaker labels (file, database, or global). Setting it to 'file'.
'Debug.SpeakerDiarization.Bug1237' found in /Users/hbredin/Development/pyannote/pyannote-audio/tests/data/database.yml does not define the 'scope' of speaker labels (file, database, or global). Setting it to 'file'.
In [2]:
from pyannote.audio.models.segmentation.debug import SimpleSegmentationModel
from pyannote.audio.models.embedding.debug import SimpleEmbeddingModel
import pytorch_lightning as pl
In [3]:
from pyannote.audio.utils.preview import listen
listen(next(protocol.train()))
Out [3]:
Your browser does not support the audio element.

Voice activity detection

In [4]:
from pyannote.audio.tasks import VoiceActivityDetection
vad = VoiceActivityDetection(protocol, duration=2., batch_size=32, num_workers=4)
model = SimpleSegmentationModel(task=vad)
trainer = pl.Trainer(max_epochs=1)
_ = trainer.fit(model)
GPU available: True (mps), used: False
Protocol Debug.SpeakerDiarization.Debug does not precompute the output of torchaudio.info(): adding a 'torchaudio.info' preprocessor for you to speed up dataloaders. See pyannote.database documentation on how to do that yourself.
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Missing logger folder: /Users/hbredin/Development/pyannote/pyannote-audio/notebook/lightning_logs

  | Name              | Type             | Params | In sizes       | Out sizes                                  
----------------------------------------------------------------------------------------------------------------------
0 | mfcc              | MFCC             | 0      | [32, 1, 32000] | [32, 1, 40, 161]                           
1 | lstm              | LSTM             | 18.9 K | [32, 161, 40]  | [[32, 161, 64], [[2, 32, 32], [2, 32, 32]]]
2 | classifier        | Linear           | 65     | [32, 161, 64]  | [32, 161, 1]                               
3 | activation        | Sigmoid          | 0      | [32, 161, 1]   | [32, 161, 1]                               
4 | validation_metric | MetricCollection | 0      | ?              | ?                                          
----------------------------------------------------------------------------------------------------------------------
19.0 K    Trainable params
0         Non-trainable params
19.0 K    Total params
0.076     Total estimated model params size (MB)
Sanity Checking: 0it [00:00, ?it/s]
Training: 0it [00:00, ?it/s]
Validation: 0it [00:00, ?it/s]
`Trainer.fit` stopped: `max_epochs=1` reached.

Overlapped speech detection

In [5]:
from pyannote.audio.tasks import OverlappedSpeechDetection
ovl = OverlappedSpeechDetection(protocol, duration=2., batch_size=32, num_workers=4)
model = SimpleSegmentationModel(task=ovl)
trainer = pl.Trainer(max_epochs=1)
_ = trainer.fit(model)
GPU available: True (mps), used: False
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs

  | Name              | Type             | Params | In sizes       | Out sizes                                  
----------------------------------------------------------------------------------------------------------------------
0 | mfcc              | MFCC             | 0      | [32, 1, 32000] | [32, 1, 40, 161]                           
1 | lstm              | LSTM             | 18.9 K | [32, 161, 40]  | [[32, 161, 64], [[2, 32, 32], [2, 32, 32]]]
2 | classifier        | Linear           | 65     | [32, 161, 64]  | [32, 161, 1]                               
3 | activation        | Sigmoid          | 0      | [32, 161, 1]   | [32, 161, 1]                               
4 | validation_metric | MetricCollection | 0      | ?              | ?                                          
----------------------------------------------------------------------------------------------------------------------
19.0 K    Trainable params
0         Non-trainable params
19.0 K    Total params
0.076     Total estimated model params size (MB)
Sanity Checking: 0it [00:00, ?it/s]
Training: 0it [00:00, ?it/s]
Validation: 0it [00:00, ?it/s]
`Trainer.fit` stopped: `max_epochs=1` reached.

(Local) speaker diarization

In [6]:
from pyannote.audio.tasks import SpeakerDiarization
seg = SpeakerDiarization(protocol, duration=2., batch_size=32, num_workers=4)
model = SimpleSegmentationModel(task=seg)
trainer = pl.Trainer(max_epochs=1)
_ = trainer.fit(model)
GPU available: True (mps), used: False
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Output()


  | Name              | Type             | Params | In sizes       | Out sizes                                  
----------------------------------------------------------------------------------------------------------------------
0 | mfcc              | MFCC             | 0      | [32, 1, 32000] | [32, 1, 40, 161]                           
1 | lstm              | LSTM             | 18.9 K | [32, 161, 40]  | [[32, 161, 64], [[2, 32, 32], [2, 32, 32]]]
2 | classifier        | Linear           | 195    | [32, 161, 64]  | [32, 161, 3]                               
3 | activation        | Sigmoid          | 0      | [32, 161, 3]   | [32, 161, 3]                               
4 | validation_metric | MetricCollection | 0      | ?              | ?                                          
----------------------------------------------------------------------------------------------------------------------
19.1 K    Trainable params
0         Non-trainable params
19.1 K    Total params
0.077     Total estimated model params size (MB)
   -  27.32% of all chunks contain no speech at all.
   -  66.79% contain 1 speaker or less
   -  91.25% contain 2 speakers or less
   -  98.04% contain 3 speakers or less
   - 100.00% contain 4 speakers or less
Setting `max_speakers_per_chunk` to 3. You can override this value (or avoid this estimation step) by passing `max_speakers_per_chunk=3` to the task constructor.
Sanity Checking: 0it [00:00, ?it/s]
Training: 0it [00:00, ?it/s]
Validation: 0it [00:00, ?it/s]
`Trainer.fit` stopped: `max_epochs=1` reached.

Speaker embedding

In [7]:
from pyannote.audio.tasks import SpeakerEmbedding
emb = SpeakerEmbedding(protocol, duration=2., num_workers=4)
model = SimpleEmbeddingModel(task=emb)
trainer = pl.Trainer(max_epochs=1)
_ = trainer.fit(model)
GPU available: True (mps), used: False
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Loading Debug.SpeakerDiarization.Debug training labels: 10file [00:00, 363.55file/s]

  | Name              | Type             | Params | In sizes       | Out sizes                                  
----------------------------------------------------------------------------------------------------------------------
0 | mfcc              | MFCC             | 0      | [32, 1, 32000] | [32, 1, 40, 161]                           
1 | lstm              | LSTM             | 18.9 K | [32, 161, 40]  | [[32, 161, 64], [[2, 32, 32], [2, 32, 32]]]
2 | loss_func         | ArcFaceLoss      | 832    | ?              | ?                                          
3 | validation_metric | MetricCollection | 0      | ?              | ?                                          
----------------------------------------------------------------------------------------------------------------------
19.8 K    Trainable params
0         Non-trainable params
19.8 K    Total params
0.079     Total estimated model params size (MB)
Sanity Checking: 0it [00:00, ?it/s]
Training: 0it [00:00, ?it/s]
`Trainer.fit` stopped: `max_epochs=1` reached.