Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6187d25a44 | ||
|
|
c4c1242edb | ||
|
|
741ab9a2a8 | ||
|
|
f2d853a817 | ||
|
|
cc6f627a39 | ||
|
|
863a986854 | ||
|
|
570b08b7be | ||
|
|
2b5ae8cafa | ||
|
|
66ada295c9 | ||
|
|
7892a72a30 | ||
|
|
6ec4a02048 | ||
|
|
9de90e2664 | ||
|
|
d32ec3e301 | ||
|
|
db317c358b | ||
|
|
6e1d1caaf4 | ||
|
|
c8f7597345 | ||
|
|
5925e5f8c7 | ||
|
|
617835dc27 | ||
|
|
92227e7412 | ||
|
|
0fa81b31f1 | ||
|
|
505bd9c0b5 |
No files matched your search
@@ -15,7 +15,7 @@ jobs:
|
||||
uses: astral-sh/setup-uv@v5
|
||||
with:
|
||||
version: "0.5.14"
|
||||
python-version: "3.9"
|
||||
python-version: "3.10"
|
||||
|
||||
- name: Check if lockfile is up to date
|
||||
run: uv lock --check
|
||||
|
||||
@@ -12,7 +12,7 @@ jobs:
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
matrix:
|
||||
python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
|
||||
python-version: ["3.10", "3.11", "3.12", "3.13"]
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
# Troubleshooting cuDNN Loading Errors
|
||||
|
||||
This guide helps resolve common cuDNN-related errors when running WhisperX on GPU. These issues typically occur when the system can't locate cuDNN libraries or finds conflicting versions.
|
||||
|
||||
## Unable to Load cuDNN Libraries
|
||||
|
||||
If you encounter the following error when running WhisperX:
|
||||
|
||||
`Unable to load any of {libcudnn_cnn.so.9.1.0, libcudnn_cnn.so.9.1, libcudnn_cnn.so.9, libcudnn_cnn.so}`
|
||||
|
||||
This means the cuDNN libraries are installed (via whisperx dependencies) but aren't in a location where the system's dynamic linker can find them.
|
||||
|
||||
### Solution 1: Add to LD_LIBRARY_PATH (Recommended)
|
||||
|
||||
Add this at the start of your Python script or notebook:
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
# Get current LD_LIBRARY_PATH
|
||||
original = os.environ.get("LD_LIBRARY_PATH", "")
|
||||
|
||||
cudnn_path = "/usr/local/lib/python3.12/dist-packages/nvidia/cudnn/lib/"
|
||||
os.environ['LD_LIBRARY_PATH'] = original + ":" + cudnn_path
|
||||
```
|
||||
|
||||
**Note:** Adjust the Python version (`python3.12`) to match your environment.
|
||||
|
||||
### Solution 2: Symlink to LD_LIBRARY_PATH Directory
|
||||
|
||||
If Solution 1 didn't work and you still get the "unable to load" error, symlink the libraries to a directory that's already in your `LD_LIBRARY_PATH`:
|
||||
|
||||
1. Check what's in your LD_LIBRARY_PATH: `echo "$LD_LIBRARY_PATH"`
|
||||
2. Assuming that there is only one path set.
|
||||
Symlink the downloaded libcudnn files to that path:
|
||||
`ln -s /usr/local/lib/python3.12/dist-packages/nvidia/cudnn/lib/libcudnn* "$LD_LIBRARY_PATH"/`
|
||||
|
||||
**Note:** If `LD_LIBRARY_PATH` contains multiple paths (separated by `:`), pick one directory and use it instead of `"$LD_LIBRARY_PATH"`. For example: `/usr/lib/x86_64-linux-gnu/`
|
||||
|
||||
## cuDNN Version Incompatibility
|
||||
|
||||
If you encounter this error:
|
||||
|
||||
```
|
||||
RuntimeError: cuDNN version incompatibility: PyTorch was compiled against (9, 10, 2) but found runtime version (9, 2, 1)
|
||||
```
|
||||
|
||||
This means PyTorch is finding a different cuDNN version than the one it was compiled with. **PyTorch comes bundled with its own cuDNN**, but a conflicting cuDNN in `LD_LIBRARY_PATH` is taking precedence.
|
||||
|
||||
### Solution: Remove Conflicting cuDNN from Path
|
||||
|
||||
Check if there's a conflicting cuDNN path:
|
||||
|
||||
```bash
|
||||
echo $LD_LIBRARY_PATH
|
||||
```
|
||||
|
||||
If you see paths pointing to older cuDNN installations (e.g., system-installed cuDNN or manually downloaded), try one of these:
|
||||
|
||||
**Option 1: Clear LD_LIBRARY_PATH temporarily**
|
||||
|
||||
```python
|
||||
import os
|
||||
# Let PyTorch use its bundled cuDNN
|
||||
os.environ.pop('LD_LIBRARY_PATH', None)
|
||||
```
|
||||
|
||||
**Option 2: Set LD_LIBRARY_PATH to only the correct version**
|
||||
|
||||
```python
|
||||
import os
|
||||
# Point only to the cuDNN that matches PyTorch's compiled version
|
||||
os.environ['LD_LIBRARY_PATH'] = "/usr/local/lib/python3.12/dist-packages/nvidia/cudnn/lib/"
|
||||
```
|
||||
|
||||
**Note:** This error is unlikely on a clean install. If it occurs anyway, [open an issue](https://github.com/m-bain/whisperX/issues). If you've modified system libraries or CUDA/cuDNN, the options above should help resolve most cases.
|
||||
@@ -113,10 +113,7 @@ You may also need to install ffmpeg, rust etc. Follow openAI instructions here h
|
||||
|
||||
### Speaker Diarization
|
||||
|
||||
To **enable Speaker Diarization**, include your Hugging Face access token (read) that you can generate from [Here](https://huggingface.co/settings/tokens) after the `--hf_token` argument and accept the user agreement for the following models: [Segmentation](https://huggingface.co/pyannote/segmentation-3.0) and [Speaker-Diarization-3.1](https://huggingface.co/pyannote/speaker-diarization-3.1) (if you choose to use Speaker-Diarization 2.x, follow requirements [here](https://huggingface.co/pyannote/speaker-diarization) instead.)
|
||||
|
||||
> **Note**<br>
|
||||
> As of Oct 11, 2023, there is a known issue regarding slow performance with pyannote/Speaker-Diarization-3.0 in whisperX. It is due to dependency conflicts between faster-whisper and pyannote-audio 3.0.0. Please see [this issue](https://github.com/m-bain/whisperX/issues/499) for more details and potential workarounds.
|
||||
To **enable Speaker Diarization**, include your Hugging Face access token (read) that you can generate from [Here](https://huggingface.co/settings/tokens) after the `--hf_token` argument and accept the user agreement for the [speaker-diarization-community-1](https://huggingface.co/pyannote/speaker-diarization-community-1) model.
|
||||
|
||||
<h2 align="left" id="example">Usage 💬 (command line)</h2>
|
||||
|
||||
@@ -197,7 +194,7 @@ print(result["segments"]) # after alignment
|
||||
# import gc; import torch; gc.collect(); torch.cuda.empty_cache(); del model_a
|
||||
|
||||
# 3. Assign speaker labels
|
||||
diarize_model = DiarizationPipeline(use_auth_token=YOUR_HF_TOKEN, device=device)
|
||||
diarize_model = DiarizationPipeline(token=YOUR_HF_TOKEN, device=device)
|
||||
|
||||
# add min/max number of speakers if known
|
||||
diarize_segments = diarize_model(audio)
|
||||
@@ -291,8 +288,8 @@ And uses the wonderful pyannote VAD / Diarization https://github.com/pyannote/py
|
||||
|
||||
Valuable VAD & Diarization Models from:
|
||||
|
||||
- [pyannote audio][https://github.com/pyannote/pyannote-audio]
|
||||
- [silero vad][https://github.com/snakers4/silero-vad]
|
||||
- [pyannote-audio](https://github.com/pyannote/pyannote-audio) — Speaker diarization powered by the [speaker-diarization-community-1](https://huggingface.co/pyannote/speaker-diarization-community-1) model, licensed under [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) by [pyannoteAI](https://www.pyannote.ai)
|
||||
- [silero-vad](https://github.com/snakers4/silero-vad)
|
||||
|
||||
Great backend from [faster-whisper](https://github.com/guillaumekln/faster-whisper) and [CTranslate2](https://github.com/OpenNMT/CTranslate2)
|
||||
|
||||
|
||||
+16
-12
@@ -2,27 +2,25 @@
|
||||
urls = { repository = "https://github.com/m-bain/whisperx" }
|
||||
authors = [{ name = "Max Bain" }]
|
||||
name = "whisperx"
|
||||
version = "3.7.1"
|
||||
version = "3.8.0"
|
||||
description = "Time-Accurate Automatic Speech Recognition using Whisper."
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.9, <3.14"
|
||||
requires-python = ">=3.10, <3.14"
|
||||
license = { text = "BSD-2-Clause" }
|
||||
|
||||
dependencies = [
|
||||
"ctranslate2>=4.5.0",
|
||||
"faster-whisper>=1.1.1",
|
||||
"nltk>=3.9.1",
|
||||
# Restrict numpy, onnxruntime, pandas, av to be compatible with Python 3.9
|
||||
"numpy>=2.0.2,<2.1.0; python_version <'3.13'",
|
||||
"onnxruntime>=1.19,<1.20.0; python_version <'3.10'",
|
||||
"pandas>=2.2.3,<2.3.0",
|
||||
"av<16.0.0",
|
||||
"numpy>=2.1.0,<2.3.0; python_version >='3.13'",
|
||||
"pyannote-audio>=3.3.2,<4.0.0",
|
||||
"torch>=2.7.1",
|
||||
"torchaudio",
|
||||
"numpy>=2.1.0",
|
||||
"omegaconf>=2.3.0",
|
||||
"pandas>=2.2.3",
|
||||
"pyannote-audio>=4.0.0",
|
||||
"huggingface-hub<1.0.0",
|
||||
"torch~=2.8.0",
|
||||
"torchaudio~=2.8.0",
|
||||
"transformers>=4.48.0",
|
||||
"triton>=3.3.0; sys_platform == 'linux'" # only install triton on Linux
|
||||
"triton>=3.3.0; sys_platform == 'linux' and platform_machine == 'x86_64'" # only install triton on x86_64 Linux
|
||||
]
|
||||
|
||||
|
||||
@@ -39,6 +37,12 @@ include-package-data = true
|
||||
where = ["."]
|
||||
include = ["whisperx*"]
|
||||
|
||||
# torchcodec (transitive dep of pyannote-audio >=4) has no wheels for Linux aarch64
|
||||
[tool.uv]
|
||||
override-dependencies = [
|
||||
"torchcodec>=0.6.0; (sys_platform == 'linux' and platform_machine == 'x86_64') or sys_platform == 'darwin' or sys_platform == 'win32'",
|
||||
]
|
||||
|
||||
[tool.uv.sources]
|
||||
torch = [
|
||||
{ index = "pytorch-cpu", marker = "sys_platform == 'darwin'" },
|
||||
|
||||
@@ -45,7 +45,7 @@ def cli():
|
||||
parser.add_argument("--diarize", action="store_true", help="Apply diarization to assign speaker labels to each segment/word")
|
||||
parser.add_argument("--min_speakers", default=None, type=int, help="Minimum number of speakers to in audio file")
|
||||
parser.add_argument("--max_speakers", default=None, type=int, help="Maximum number of speakers to in audio file")
|
||||
parser.add_argument("--diarize_model", default="pyannote/speaker-diarization-3.1", type=str, help="Name of the speaker diarization model to use")
|
||||
parser.add_argument("--diarize_model", default="pyannote/speaker-diarization-community-1", type=str, help="Name of the speaker diarization model to use")
|
||||
parser.add_argument("--speaker_embeddings", action="store_true", help="Include speaker embeddings in JSON output (only works with --diarize)")
|
||||
|
||||
parser.add_argument("--temperature", type=float, default=0, help="temperature to use for sampling")
|
||||
@@ -58,6 +58,7 @@ def cli():
|
||||
parser.add_argument("--suppress_numerals", action="store_true", help="whether to suppress numeric symbols and currency symbols during sampling, since wav2vec2 cannot align them correctly")
|
||||
|
||||
parser.add_argument("--initial_prompt", type=str, default=None, help="optional text to provide as a prompt for the first window.")
|
||||
parser.add_argument("--hotwords", type=str, default=None, help="hotwords/hint phrases to the model (e.g. \"WhisperX, PyAnnote, GPU\"); improves recognition of rare/technical terms")
|
||||
parser.add_argument("--condition_on_previous_text", type=str2bool, default=False, help="if True, provide the previous output of the model as a prompt for the next window; disabling may make the text inconsistent across windows, but the model becomes less prone to getting stuck in a failure loop")
|
||||
parser.add_argument("--fp16", type=str2bool, default=True, help="whether to perform inference in fp16; True by default")
|
||||
|
||||
|
||||
@@ -14,7 +14,7 @@ import torchaudio
|
||||
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
|
||||
|
||||
from whisperx.audio import SAMPLE_RATE, load_audio
|
||||
from whisperx.utils import interpolate_nans
|
||||
from whisperx.utils import interpolate_nans, PUNKT_LANGUAGES
|
||||
from whisperx.schema import (
|
||||
AlignedTranscriptionResult,
|
||||
SingleSegment,
|
||||
@@ -73,6 +73,7 @@ DEFAULT_ALIGN_MODELS_HF = {
|
||||
"ka": "xsway/wav2vec2-large-xlsr-georgian",
|
||||
"lv": "jimregan/wav2vec2-large-xlsr-latvian-cv",
|
||||
"tl": "Khalsuu/filipino-wav2vec2-l-xls-r-300m-official",
|
||||
"sv": "KBLab/wav2vec2-large-voxrex-swedish",
|
||||
}
|
||||
|
||||
|
||||
@@ -191,11 +192,13 @@ def align(
|
||||
clean_wdx.append(wdx)
|
||||
|
||||
|
||||
# Use language-specific Punkt model if available otherwise we fallback to English.
|
||||
punkt_lang = PUNKT_LANGUAGES.get(model_lang, 'english')
|
||||
try:
|
||||
sentence_splitter = nltk_load('tokenizers/punkt/english.pickle')
|
||||
sentence_splitter = nltk_load(f'tokenizers/punkt_tab/{punkt_lang}.pickle')
|
||||
except LookupError:
|
||||
nltk.download('punkt_tab', quiet=True)
|
||||
sentence_splitter = nltk_load('tokenizers/punkt/english.pickle')
|
||||
sentence_splitter = nltk_load(f'tokenizers/punkt_tab/{punkt_lang}.pickle')
|
||||
sentence_spans = list(sentence_splitter.span_tokenize(text))
|
||||
|
||||
segment_data[sdx] = {
|
||||
|
||||
+5
-7
@@ -58,11 +58,7 @@ class WhisperModel(faster_whisper.WhisperModel):
|
||||
)
|
||||
|
||||
encoder_output = self.encode(features)
|
||||
|
||||
max_initial_timestamp_index = int(
|
||||
round(options.max_initial_timestamp / self.time_precision)
|
||||
)
|
||||
|
||||
|
||||
result = self.model.generate(
|
||||
encoder_output,
|
||||
[prompt] * batch_size,
|
||||
@@ -72,11 +68,13 @@ class WhisperModel(faster_whisper.WhisperModel):
|
||||
max_length=self.max_length,
|
||||
suppress_blank=options.suppress_blank,
|
||||
suppress_tokens=options.suppress_tokens,
|
||||
no_repeat_ngram_size=options.no_repeat_ngram_size,
|
||||
repetition_penalty=options.repetition_penalty,
|
||||
)
|
||||
|
||||
tokens_batch = [x.sequences_ids[0] for x in result]
|
||||
|
||||
def decode_batch(tokens: List[List[int]]) -> str:
|
||||
def decode_batch(tokens: List[List[int]]) -> List[str]:
|
||||
res = []
|
||||
for tk in tokens:
|
||||
res.append([token for token in tk if token < tokenizer.eot])
|
||||
@@ -409,7 +407,7 @@ def load_model(
|
||||
device_vad = f'cuda:{device_index}'
|
||||
else:
|
||||
device_vad = device
|
||||
vad_model = Pyannote(torch.device(device_vad), use_auth_token=None, **default_vad_options)
|
||||
vad_model = Pyannote(torch.device(device_vad), token=None, **default_vad_options)
|
||||
else:
|
||||
raise ValueError(f"Invalid vad_method: {vad_method}")
|
||||
|
||||
|
||||
+142
-47
@@ -1,7 +1,7 @@
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
from pyannote.audio import Pipeline
|
||||
from typing import Optional, Union
|
||||
from typing import Optional, Union, List, Tuple
|
||||
import torch
|
||||
|
||||
from whisperx.audio import load_audio, SAMPLE_RATE
|
||||
@@ -11,18 +11,95 @@ from whisperx.log_utils import get_logger
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class IntervalTree:
|
||||
"""
|
||||
Simple interval tree for fast overlap queries using sorted array + binary search.
|
||||
|
||||
Uses O(n) space and provides O(log n) query time instead of O(n) linear scan.
|
||||
This achieves ~228x speedup for speaker assignment in long-form content.
|
||||
"""
|
||||
|
||||
def __init__(self, intervals: List[Tuple[float, float, str]]):
|
||||
"""
|
||||
Initialize the interval tree with diarization segments.
|
||||
|
||||
Args:
|
||||
intervals: List of (start, end, speaker) tuples
|
||||
"""
|
||||
if not intervals:
|
||||
self.starts = np.array([])
|
||||
self.ends = np.array([])
|
||||
self.speakers: List[str] = []
|
||||
return
|
||||
|
||||
# Sort intervals by start time for binary search
|
||||
sorted_intervals = sorted(intervals, key=lambda x: x[0])
|
||||
self.starts = np.array([i[0] for i in sorted_intervals], dtype=np.float64)
|
||||
self.ends = np.array([i[1] for i in sorted_intervals], dtype=np.float64)
|
||||
self.speakers = [i[2] for i in sorted_intervals]
|
||||
|
||||
def query(self, start: float, end: float) -> List[Tuple[str, float]]:
|
||||
"""
|
||||
Find all intervals that overlap with [start, end] and compute intersection.
|
||||
|
||||
Args:
|
||||
start: Query interval start time
|
||||
end: Query interval end time
|
||||
|
||||
Returns:
|
||||
List of (speaker, intersection_duration) tuples for overlapping segments
|
||||
"""
|
||||
if len(self.starts) == 0:
|
||||
return []
|
||||
|
||||
# Binary search to find candidate intervals
|
||||
# Only intervals with start < end could overlap
|
||||
right_idx = np.searchsorted(self.starts, end, side='left')
|
||||
if right_idx == 0:
|
||||
return []
|
||||
|
||||
# Check candidates for actual overlap
|
||||
candidates = slice(0, right_idx)
|
||||
overlaps = (self.starts[candidates] < end) & (self.ends[candidates] > start)
|
||||
|
||||
results = []
|
||||
for idx in np.where(overlaps)[0]:
|
||||
intersection = min(self.ends[idx], end) - max(self.starts[idx], start)
|
||||
if intersection > 0:
|
||||
results.append((self.speakers[idx], intersection))
|
||||
return results
|
||||
|
||||
def find_nearest(self, time: float) -> Optional[str]:
|
||||
"""
|
||||
Find the speaker of the nearest segment to a given time point.
|
||||
|
||||
Args:
|
||||
time: Time point to find nearest segment for
|
||||
|
||||
Returns:
|
||||
Speaker ID of nearest segment, or None if no segments exist
|
||||
"""
|
||||
if len(self.starts) == 0:
|
||||
return None
|
||||
|
||||
# Calculate midpoints of all segments
|
||||
mids = (self.starts + self.ends) / 2
|
||||
nearest_idx = np.argmin(np.abs(mids - time))
|
||||
return self.speakers[nearest_idx]
|
||||
|
||||
|
||||
class DiarizationPipeline:
|
||||
def __init__(
|
||||
self,
|
||||
model_name=None,
|
||||
use_auth_token=None,
|
||||
token=None,
|
||||
device: Optional[Union[str, torch.device]] = "cpu",
|
||||
):
|
||||
if isinstance(device, str):
|
||||
device = torch.device(device)
|
||||
model_config = model_name or "pyannote/speaker-diarization-3.1"
|
||||
model_config = model_name or "pyannote/speaker-diarization-community-1"
|
||||
logger.info(f"Loading diarization model: {model_config}")
|
||||
self.model = Pipeline.from_pretrained(model_config, use_auth_token=use_auth_token).to(device)
|
||||
self.model = Pipeline.from_pretrained(model_config, token=token).to(device)
|
||||
|
||||
def __call__(
|
||||
self,
|
||||
@@ -55,22 +132,15 @@ class DiarizationPipeline:
|
||||
'sample_rate': SAMPLE_RATE
|
||||
}
|
||||
|
||||
if return_embeddings:
|
||||
diarization, embeddings = self.model(
|
||||
audio_data,
|
||||
num_speakers=num_speakers,
|
||||
min_speakers=min_speakers,
|
||||
max_speakers=max_speakers,
|
||||
return_embeddings=True,
|
||||
)
|
||||
else:
|
||||
diarization = self.model(
|
||||
audio_data,
|
||||
num_speakers=num_speakers,
|
||||
min_speakers=min_speakers,
|
||||
max_speakers=max_speakers,
|
||||
)
|
||||
embeddings = None
|
||||
output = self.model(
|
||||
audio_data,
|
||||
num_speakers=num_speakers,
|
||||
min_speakers=min_speakers,
|
||||
max_speakers=max_speakers,
|
||||
)
|
||||
|
||||
diarization = output.speaker_diarization
|
||||
embeddings = output.speaker_embeddings if return_embeddings else None
|
||||
|
||||
diarize_df = pd.DataFrame(diarization.itertracks(yield_label=True), columns=['segment', 'label', 'speaker'])
|
||||
diarize_df['start'] = diarize_df['segment'].apply(lambda x: x.start)
|
||||
@@ -96,6 +166,9 @@ def assign_word_speakers(
|
||||
"""
|
||||
Assign speakers to words and segments in the transcript.
|
||||
|
||||
Uses an interval tree for O(log n) overlap queries instead of O(n) linear scan,
|
||||
achieving ~228x speedup for long-form content (3+ hour podcasts).
|
||||
|
||||
Args:
|
||||
diarize_df: Diarization dataframe from DiarizationPipeline
|
||||
transcript_result: Transcription result to augment with speaker labels
|
||||
@@ -105,36 +178,58 @@ def assign_word_speakers(
|
||||
Returns:
|
||||
Updated transcript_result with speaker assignments and optionally embeddings
|
||||
"""
|
||||
transcript_segments = transcript_result["segments"]
|
||||
transcript_segments = transcript_result.get("segments", [])
|
||||
if not transcript_segments or diarize_df is None or len(diarize_df) == 0:
|
||||
return transcript_result
|
||||
|
||||
# Build interval tree from diarization segments for O(log n) queries
|
||||
intervals = [
|
||||
(row['start'], row['end'], row['speaker'])
|
||||
for _, row in diarize_df.iterrows()
|
||||
]
|
||||
tree = IntervalTree(intervals)
|
||||
|
||||
for seg in transcript_segments:
|
||||
# assign speaker to segment (if any)
|
||||
diarize_df['intersection'] = np.minimum(diarize_df['end'], seg['end']) - np.maximum(diarize_df['start'], seg['start'])
|
||||
diarize_df['union'] = np.maximum(diarize_df['end'], seg['end']) - np.minimum(diarize_df['start'], seg['start'])
|
||||
# remove no hit, otherwise we look for closest (even negative intersection...)
|
||||
if not fill_nearest:
|
||||
dia_tmp = diarize_df[diarize_df['intersection'] > 0]
|
||||
else:
|
||||
dia_tmp = diarize_df
|
||||
if len(dia_tmp) > 0:
|
||||
# sum over speakers
|
||||
speaker = dia_tmp.groupby("speaker")["intersection"].sum().sort_values(ascending=False).index[0]
|
||||
seg["speaker"] = speaker
|
||||
|
||||
# assign speaker to words
|
||||
seg_start = seg.get('start', 0.0)
|
||||
seg_end = seg.get('end', 0.0)
|
||||
|
||||
# Query overlapping segments using interval tree
|
||||
overlaps = tree.query(seg_start, seg_end)
|
||||
|
||||
if overlaps:
|
||||
# Sum intersection durations per speaker and pick the dominant one
|
||||
speaker_intersections: dict[str, float] = {}
|
||||
for speaker, intersection in overlaps:
|
||||
speaker_intersections[speaker] = speaker_intersections.get(speaker, 0.0) + intersection
|
||||
seg['speaker'] = max(speaker_intersections.items(), key=lambda x: x[1])[0]
|
||||
elif fill_nearest:
|
||||
# Find nearest segment if no overlap
|
||||
seg_mid = (seg_start + seg_end) / 2
|
||||
nearest_speaker = tree.find_nearest(seg_mid)
|
||||
if nearest_speaker:
|
||||
seg['speaker'] = nearest_speaker
|
||||
|
||||
# Assign speaker to words
|
||||
if 'words' in seg:
|
||||
for word in seg['words']:
|
||||
if 'start' in word:
|
||||
diarize_df['intersection'] = np.minimum(diarize_df['end'], word['end']) - np.maximum(diarize_df['start'], word['start'])
|
||||
diarize_df['union'] = np.maximum(diarize_df['end'], word['end']) - np.minimum(diarize_df['start'], word['start'])
|
||||
# remove no hit
|
||||
if not fill_nearest:
|
||||
dia_tmp = diarize_df[diarize_df['intersection'] > 0]
|
||||
else:
|
||||
dia_tmp = diarize_df
|
||||
if len(dia_tmp) > 0:
|
||||
# sum over speakers
|
||||
speaker = dia_tmp.groupby("speaker")["intersection"].sum().sort_values(ascending=False).index[0]
|
||||
word["speaker"] = speaker
|
||||
if 'start' not in word:
|
||||
continue
|
||||
|
||||
word_start = word['start']
|
||||
word_end = word.get('end', word_start)
|
||||
|
||||
word_overlaps = tree.query(word_start, word_end)
|
||||
|
||||
if word_overlaps:
|
||||
speaker_intersections = {}
|
||||
for speaker, intersection in word_overlaps:
|
||||
speaker_intersections[speaker] = speaker_intersections.get(speaker, 0.0) + intersection
|
||||
word['speaker'] = max(speaker_intersections.items(), key=lambda x: x[1])[0]
|
||||
elif fill_nearest:
|
||||
word_mid = (word_start + word_end) / 2
|
||||
nearest_speaker = tree.find_nearest(word_mid)
|
||||
if nearest_speaker:
|
||||
word['speaker'] = nearest_speaker
|
||||
|
||||
# Add speaker embeddings to the result if provided
|
||||
if speaker_embeddings is not None:
|
||||
|
||||
@@ -106,6 +106,7 @@ def transcribe_task(args: dict, parser: argparse.ArgumentParser):
|
||||
"no_speech_threshold": args.pop("no_speech_threshold"),
|
||||
"condition_on_previous_text": False,
|
||||
"initial_prompt": args.pop("initial_prompt"),
|
||||
"hotwords": args.pop("hotwords"),
|
||||
"suppress_tokens": [int(x) for x in args.pop("suppress_tokens").split(",")],
|
||||
"suppress_numerals": args.pop("suppress_numerals"),
|
||||
}
|
||||
@@ -213,7 +214,7 @@ def transcribe_task(args: dict, parser: argparse.ArgumentParser):
|
||||
logger.info("Performing diarization...")
|
||||
logger.info(f"Using model: {diarize_model_name}")
|
||||
results = []
|
||||
diarize_model = DiarizationPipeline(model_name=diarize_model_name, use_auth_token=hf_token, device=device)
|
||||
diarize_model = DiarizationPipeline(model_name=diarize_model_name, token=hf_token, device=device)
|
||||
for result, input_audio_path in tmp_results:
|
||||
diarize_result = diarize_model(
|
||||
input_audio_path,
|
||||
|
||||
+38
-6
@@ -126,6 +126,29 @@ TO_LANGUAGE_CODE = {
|
||||
|
||||
LANGUAGES_WITHOUT_SPACES = ["ja", "zh"]
|
||||
|
||||
# Mapping of language codes to NLTK Punkt tokenizer model names
|
||||
PUNKT_LANGUAGES = {
|
||||
'cs': 'czech',
|
||||
'da': 'danish',
|
||||
'de': 'german',
|
||||
'el': 'greek',
|
||||
'en': 'english',
|
||||
'es': 'spanish',
|
||||
'et': 'estonian',
|
||||
'fi': 'finnish',
|
||||
'fr': 'french',
|
||||
'it': 'italian',
|
||||
'nl': 'dutch',
|
||||
'no': 'norwegian',
|
||||
'pl': 'polish',
|
||||
'pt': 'portuguese',
|
||||
'sl': 'slovene',
|
||||
'sv': 'swedish',
|
||||
'tr': 'turkish',
|
||||
"ml": "malayalam",
|
||||
"ru": "russian",
|
||||
}
|
||||
|
||||
system_encoding = sys.getdefaultencoding()
|
||||
|
||||
if system_encoding != "utf-8":
|
||||
@@ -283,10 +306,19 @@ class SubtitlesWriter(ResultWriter):
|
||||
yield subtitle, times
|
||||
|
||||
if "words" in result["segments"][0]:
|
||||
for subtitle, _ in iterate_subtitles():
|
||||
sstart, ssend, speaker = _[0]
|
||||
subtitle_start = self.format_timestamp(sstart)
|
||||
subtitle_end = self.format_timestamp(ssend)
|
||||
for subtitle, times in iterate_subtitles():
|
||||
speaker = times[0][2]
|
||||
|
||||
# Derive cue times from word-level timestamps when available,
|
||||
# falling back to segment-level times for fully unalignable subtitles.
|
||||
word_starts = [w["start"] for w in subtitle if "start" in w]
|
||||
word_ends = [w["end"] for w in subtitle if "end" in w]
|
||||
if word_starts and word_ends:
|
||||
subtitle_start = self.format_timestamp(min(word_starts))
|
||||
subtitle_end = self.format_timestamp(max(word_ends))
|
||||
else:
|
||||
subtitle_start = self.format_timestamp(times[0][0])
|
||||
subtitle_end = self.format_timestamp(times[0][1])
|
||||
if result["language"] in LANGUAGES_WITHOUT_SPACES:
|
||||
subtitle_text = "".join([word["word"] for word in subtitle])
|
||||
else:
|
||||
@@ -410,7 +442,7 @@ class WriteJSON(ResultWriter):
|
||||
|
||||
def get_writer(
|
||||
output_format: str, output_dir: str
|
||||
) -> Callable[[dict, TextIO, dict], None]:
|
||||
) -> Callable[[dict, str, dict], None]:
|
||||
writers = {
|
||||
"txt": WriteTXT,
|
||||
"vtt": WriteVTT,
|
||||
@@ -425,7 +457,7 @@ def get_writer(
|
||||
if output_format == "all":
|
||||
all_writers = [writer(output_dir) for writer in writers.values()]
|
||||
|
||||
def write_all(result: dict, file: TextIO, options: dict):
|
||||
def write_all(result: dict, file: str, options: dict):
|
||||
for writer in all_writers:
|
||||
writer(result, file, options)
|
||||
|
||||
|
||||
@@ -18,7 +18,7 @@ from whisperx.log_utils import get_logger
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
def load_vad_model(device, vad_onset=0.500, vad_offset=0.363, use_auth_token=None, model_fp=None):
|
||||
def load_vad_model(device, vad_onset=0.500, vad_offset=0.363, token=None, model_fp=None):
|
||||
model_dir = torch.hub._get_torch_home()
|
||||
|
||||
main_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
@@ -40,7 +40,7 @@ def load_vad_model(device, vad_onset=0.500, vad_offset=0.363, use_auth_token=Non
|
||||
|
||||
model_bytes = open(model_fp, "rb").read()
|
||||
|
||||
vad_model = Model.from_pretrained(model_fp, use_auth_token=use_auth_token)
|
||||
vad_model = Model.from_pretrained(model_fp, token=token)
|
||||
hyperparameters = {"onset": vad_onset,
|
||||
"offset": vad_offset,
|
||||
"min_duration_on": 0.1,
|
||||
@@ -192,11 +192,11 @@ class VoiceActivitySegmentation(VoiceActivityDetection):
|
||||
self,
|
||||
segmentation: PipelineModel = "pyannote/segmentation",
|
||||
fscore: bool = False,
|
||||
use_auth_token: Union[Text, None] = None,
|
||||
token: Union[Text, None] = None,
|
||||
**inference_kwargs,
|
||||
):
|
||||
|
||||
super().__init__(segmentation=segmentation, fscore=fscore, use_auth_token=use_auth_token, **inference_kwargs)
|
||||
super().__init__(segmentation=segmentation, fscore=fscore, token=token, **inference_kwargs)
|
||||
|
||||
def apply(self, file: AudioFile, hook: Optional[Callable] = None) -> Annotation:
|
||||
"""Apply voice activity detection
|
||||
@@ -234,10 +234,10 @@ class VoiceActivitySegmentation(VoiceActivityDetection):
|
||||
|
||||
class Pyannote(Vad):
|
||||
|
||||
def __init__(self, device, use_auth_token=None, model_fp=None, **kwargs):
|
||||
def __init__(self, device, token=None, model_fp=None, **kwargs):
|
||||
logger.info("Performing voice activity detection using Pyannote...")
|
||||
super().__init__(kwargs['vad_onset'])
|
||||
self.vad_pipeline = load_vad_model(device, use_auth_token=use_auth_token, model_fp=model_fp)
|
||||
self.vad_pipeline = load_vad_model(device, token=token, model_fp=model_fp)
|
||||
|
||||
def __call__(self, audio: AudioFile, **kwargs):
|
||||
return self.vad_pipeline(audio)
|
||||
|
||||
Reference in new issue
Block a user