Compare commits

...
32 Commits
Author SHA1 Message Date
Max Bain 064f737998 fix: default compute_type to float32 on CPU to avoid float16 ValueError 2026-02-22 16:12:55 +00:00
Claude-Assistant d8a078eed4 feat: expose avg_logprob per segment from ctranslate2 beam search
Pass through the average log probability (transcription confidence score)
from ctranslate2 to the final segment output. The field is NotRequired
so existing code constructing segments without it remains valid.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-19 07:48:34 +01:00
carlito 4017efc273 Merge pull request #1356 from m-bain/revert-1355-batch_wrap
Revert "Batch wrap"
2026-02-17 13:34:20 +00:00
carlito 0e073d418c Revert "Batch wrap" 2026-02-17 13:33:06 +00:00
carlito 422c44f952 Merge pull request #1355 from 1carlito/batch_wrap
Batch wrap
2026-02-17 13:31:15 +00:00
carlos 1b6a3b78a6 [feat] First batch wrap around
[New File] benchmark testing
2026-02-17 13:30:03 +00:00
carlos de5fa652a1 [modification]
added and condition before streams, existing logic is not chnaged.
2026-02-16 15:43:01 +00:00
carlos e33bb1ec25 Although the existing commit worked, inital prompt was in the loop no need. 2026-02-16 15:43:01 +00:00
carlos 1430e43300 [fix] Batch context is updated each time.
It works with the initial prompt added.

Ran pdb to make sure and check output.

Long audio works.

Existing Logic is correct without flag.
2026-02-16 15:43:01 +00:00
Barabazs 42beab1f9f chore: bump version to 3.8.1 2026-02-14 15:00:56 +01:00
Barabazs 1baf8d2314 feat: pass --hf_token to WhisperModel for gated model support
Forward the existing --hf_token CLI argument to faster-whisper's
WhisperModel via a new use_auth_token parameter on load_model(),
enabling downloads of gated/private HuggingFace models.
2026-02-14 14:48:04 +01:00
MrPrayerandBarabazs 9d687e015e fix: propagate --model_dir and --model_cache_only to all model loading paths (#1285)
- Add `model_cache_only` param to `load_align_model()`, pass as `local_files_only` to HuggingFace `from_pretrained` calls
- Forward `model_dir` and `model_cache_only` to both `load_align_model` call sites (initial load and language-change reload)
- Add `cache_dir` param to `DiarizationPipeline.__init__`, forward to pyannote `Pipeline.from_pretrained`
- Pass `--model_dir` as `cache_dir` when constructing `DiarizationPipeline` in CLI

Previously only the ASR model respected these flags. Alignment and diarization models would always download from HuggingFace to the default cache, breaking offline and custom-cache workflows.


---------

Co-authored-by: Barabazs <31799121+Barabazs@users.noreply.github.com>
2026-02-14 14:11:52 +01:00
BarabazsandGiorgio Azzinnaro 6187d25a44 feat: migrate to pyannote-audio v4 with speaker-diarization-community-1 (#1349)
* feat: upgrade pyannote-audio dependency to v4

* fix: rename use_auth_token to token for pyannote-audio v4 compatibility

* fix: add omegaconf dep

* fix: use structured output API for pyannote-audio v4 diarization

pyannote-audio 4.x no longer returns a plain Annotation (or tuple when>
return_embeddings=True). It now returns a structured output with
speaker_diarization and speaker_embeddings attributes.

* feat: switch default diarization model to speaker-diarization-community-1

Update default from pyannote/speaker-diarization-3.1 to
pyannote/speaker-diarization-community-1 (pyannote-audio v4),
add CC-BY-4.0 attribution, and update README for v4 API changes.

* fix: correct markdown link formatting for silero-vad in README.md

* chore: update version to 3.8.0


Co-authored-by: Giorgio Azzinnaro <giorgio@azzinna.ro>
2026-02-13 13:52:32 -07:00
BarabazsandClaude Opus 4.6 c4c1242edb fix: derive SRT/VTT cue times from word-level timestamps (#1347)
* fix: derive SRT/VTT cue times from word-level timestamps (#1315)

Subtitle cue start/end times were sourced from VAD segment boundaries
instead of word-level timestamps from forced alignment. This caused cues
to appear prematurely and could produce backwards chronological ordering
when VAD segments overlap.

Use min(word starts) / max(word ends) for cue timing, falling back to
segment-level times only when all words are unalignable.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: bump version to 3.7.7 in pyproject.toml

---------

Co-authored-by: Claude-Assistant <noreply@anthropic.com>
2026-02-13 12:53:49 +01:00
carlito 741ab9a2a8 Merge pull request #1343 from m-bain/fix-type-hint-decode-batch
[BugFix] Type hint fix in decode_batch List[str] not str:
2026-02-10 14:46:45 +00:00
carlito f2d853a817 Merge pull request #1342 from 1carlito/bugs
[BugFix] The variable I removed was not being used anyhwere.
2026-02-10 14:39:14 +00:00
carlos cc6f627a39 [BugFix] The variable I removed was not being used anyhwere. 2026-02-10 14:38:13 +00:00
Max Bain 863a986854 Merge pull request #1340 from RickSanchez93/main
Fix: pass no_repeat_ngram_size and repetition_penalty to CTranslate2 generate()
2026-02-10 13:02:21 +00:00
p.sacco 570b08b7be fix: add no_repeat_ngram_size and repetition_penalty options to WhisperModel 2026-02-10 10:44:58 +00:00
carlos 2b5ae8cafa [BugFix] Type hint fix in decode_batch List[str] not str: 2026-02-09 09:55:29 +00:00
Max Bain 66ada295c9 Merge pull request #1338 from Mr-Neutr0n/perf/interval-tree-speaker-assignment
Optimize assign_word_speakers with interval tree for 228x speedup
2026-02-08 18:21:32 +00:00
Mr-Neutr0n 7892a72a30 Optimize assign_word_speakers with interval tree for 228x speedup
Replace O(n*m) pandas operations with O(n log m) interval tree queries
for speaker assignment, where n = words/segments and m = diarization segments.

Performance improvement:
- 7-minute video (1185 words, 147 segments): 73.9s -> 0.32s (228x faster)
- 3-hour podcast: Minutes of processing -> Seconds

Changes:
- Add IntervalTree class using sorted array + binary search
- Refactor assign_word_speakers to use interval tree for overlap queries
- Maintain backward compatibility with same function signature
- Identical output to original implementation

The interval tree uses numpy arrays for efficient storage and binary search
(np.searchsorted) for O(log n) candidate finding, then filters candidates
for actual overlaps.

Fixes #1335
2026-02-07 12:55:26 +05:30
Barabazs 6ec4a02048 chore: drop python 3.9 support (#1328)
* chore: drop python 3.9 support

- Update requires-python to >=3.10
- Remove onnxruntime constraint (only needed for 3.9)
- Simplify numpy (remove version markers and upper bound)
- Remove pandas upper bound (<2.3.0 was for 3.9 compat)
- Remove av direct dependency (transitive via faster-whisper)

* chore(ci): remove python 3.9 from workflows

- Update build-and-release to use Python 3.10
- Remove 3.9 from python-compatibility matrix

* chore: bump version to 3.7.6
2026-01-27 10:41:40 +01:00
Barabazs 9de90e2664 fix: pin huggingface-hub<1.0.0 for pyannote-audio compatibility (#1327)
* fix: pin huggingface-hub<1.0.0 for pyannote-audio compatibility

pyannote-audio uses the deprecated `use_auth_token` parameter which was removed in huggingface-hub v1.0.0

* fix: upgrade yanked dependencies

* chore: update version to 3.7.5
2026-01-27 09:45:05 +01:00
Barabazs d32ec3e301 fix: add missing comma 2025-10-21 09:13:50 -06:00
db317c358b feat: add language-aware sentence tokenization (#1269)
* feat: add language-aware sentence tokenization

* feat: add missing punkt languages

---------

Co-authored-by: pulkit <129310466+p1kit@users.noreply.github.com>
Co-authored-by: Barabazs <31799121+Barabazs@users.noreply.github.com>
2025-10-21 15:57:26 +02:00
JulianFP 6e1d1caaf4 fix: incorrect type annotation in get_writer return value
The audio_path attribute that the __call__ method of the ResultWriter class takes is a str, not TextIO
2025-10-17 09:43:43 -06:00
Barabazs c8f7597345 feat: add hotwords argument to CLI for improved recognition of rare terms 2025-10-17 09:21:56 -06:00
Barabazs 5925e5f8c7 docs: add cuDNN troubleshooting for common issues (#1266)
* docs: add troubleshooting guide for cuDNN loading errors

* docs: add cuDNN version incompatibility troubleshooting
2025-10-16 10:56:51 +02:00
Barabazs 617835dc27 chore: upgrade torch and torchaudio dependencies to 2.8.0 2025-10-16 07:41:45 +00:00
Barabazs 92227e7412 fix: lock down torch and torchaudio versions (#1265)
* fix: update torch and torchaudio dependencies to use compatible version specifiers

* chore: update version to 3.7.3
2025-10-16 08:42:10 +02:00
PaffeandPaffe 0fa81b31f1 feat: add Swedish alignment model (#1110)
---------

Co-authored-by: Paffe <paffe@Sigge.home>
2025-10-15 08:08:18 +02:00
14 changed files with 770 additions and 1527 deletions

No files matched your search

+1 -1
View File
@@ -15,7 +15,7 @@ jobs:
uses: astral-sh/setup-uv@v5
with:
version: "0.5.14"
python-version: "3.9"
python-version: "3.10"
- name: Check if lockfile is up to date
run: uv lock --check
+1 -1
View File
@@ -12,7 +12,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
python-version: ["3.10", "3.11", "3.12", "3.13"]
steps:
- uses: actions/checkout@v4
+76
View File
@@ -0,0 +1,76 @@
# Troubleshooting cuDNN Loading Errors
This guide helps resolve common cuDNN-related errors when running WhisperX on GPU. These issues typically occur when the system can't locate cuDNN libraries or finds conflicting versions.
## Unable to Load cuDNN Libraries
If you encounter the following error when running WhisperX:
`Unable to load any of {libcudnn_cnn.so.9.1.0, libcudnn_cnn.so.9.1, libcudnn_cnn.so.9, libcudnn_cnn.so}`
This means the cuDNN libraries are installed (via whisperx dependencies) but aren't in a location where the system's dynamic linker can find them.
### Solution 1: Add to LD_LIBRARY_PATH (Recommended)
Add this at the start of your Python script or notebook:
```python
import os
# Get current LD_LIBRARY_PATH
original = os.environ.get("LD_LIBRARY_PATH", "")
cudnn_path = "/usr/local/lib/python3.12/dist-packages/nvidia/cudnn/lib/"
os.environ['LD_LIBRARY_PATH'] = original + ":" + cudnn_path
```
**Note:** Adjust the Python version (`python3.12`) to match your environment.
### Solution 2: Symlink to LD_LIBRARY_PATH Directory
If Solution 1 didn't work and you still get the "unable to load" error, symlink the libraries to a directory that's already in your `LD_LIBRARY_PATH`:
1. Check what's in your LD_LIBRARY_PATH: `echo "$LD_LIBRARY_PATH"`
2. Assuming that there is only one path set.
Symlink the downloaded libcudnn files to that path:
`ln -s /usr/local/lib/python3.12/dist-packages/nvidia/cudnn/lib/libcudnn* "$LD_LIBRARY_PATH"/`
**Note:** If `LD_LIBRARY_PATH` contains multiple paths (separated by `:`), pick one directory and use it instead of `"$LD_LIBRARY_PATH"`. For example: `/usr/lib/x86_64-linux-gnu/`
## cuDNN Version Incompatibility
If you encounter this error:
```
RuntimeError: cuDNN version incompatibility: PyTorch was compiled against (9, 10, 2) but found runtime version (9, 2, 1)
```
This means PyTorch is finding a different cuDNN version than the one it was compiled with. **PyTorch comes bundled with its own cuDNN**, but a conflicting cuDNN in `LD_LIBRARY_PATH` is taking precedence.
### Solution: Remove Conflicting cuDNN from Path
Check if there's a conflicting cuDNN path:
```bash
echo $LD_LIBRARY_PATH
```
If you see paths pointing to older cuDNN installations (e.g., system-installed cuDNN or manually downloaded), try one of these:
**Option 1: Clear LD_LIBRARY_PATH temporarily**
```python
import os
# Let PyTorch use its bundled cuDNN
os.environ.pop('LD_LIBRARY_PATH', None)
```
**Option 2: Set LD_LIBRARY_PATH to only the correct version**
```python
import os
# Point only to the cuDNN that matches PyTorch's compiled version
os.environ['LD_LIBRARY_PATH'] = "/usr/local/lib/python3.12/dist-packages/nvidia/cudnn/lib/"
```
**Note:** This error is unlikely on a clean install. If it occurs anyway, [open an issue](https://github.com/m-bain/whisperX/issues). If you've modified system libraries or CUDA/cuDNN, the options above should help resolve most cases.
+4 -7
View File
@@ -113,10 +113,7 @@ You may also need to install ffmpeg, rust etc. Follow openAI instructions here h
### Speaker Diarization
To **enable Speaker Diarization**, include your Hugging Face access token (read) that you can generate from [Here](https://huggingface.co/settings/tokens) after the `--hf_token` argument and accept the user agreement for the following models: [Segmentation](https://huggingface.co/pyannote/segmentation-3.0) and [Speaker-Diarization-3.1](https://huggingface.co/pyannote/speaker-diarization-3.1) (if you choose to use Speaker-Diarization 2.x, follow requirements [here](https://huggingface.co/pyannote/speaker-diarization) instead.)
> **Note**<br>
> As of Oct 11, 2023, there is a known issue regarding slow performance with pyannote/Speaker-Diarization-3.0 in whisperX. It is due to dependency conflicts between faster-whisper and pyannote-audio 3.0.0. Please see [this issue](https://github.com/m-bain/whisperX/issues/499) for more details and potential workarounds.
To **enable Speaker Diarization**, include your Hugging Face access token (read) that you can generate from [Here](https://huggingface.co/settings/tokens) after the `--hf_token` argument and accept the user agreement for the [speaker-diarization-community-1](https://huggingface.co/pyannote/speaker-diarization-community-1) model.
<h2 align="left" id="example">Usage 💬 (command line)</h2>
@@ -197,7 +194,7 @@ print(result["segments"]) # after alignment
# import gc; import torch; gc.collect(); torch.cuda.empty_cache(); del model_a
# 3. Assign speaker labels
diarize_model = DiarizationPipeline(use_auth_token=YOUR_HF_TOKEN, device=device)
diarize_model = DiarizationPipeline(token=YOUR_HF_TOKEN, device=device)
# add min/max number of speakers if known
diarize_segments = diarize_model(audio)
@@ -291,8 +288,8 @@ And uses the wonderful pyannote VAD / Diarization https://github.com/pyannote/py
Valuable VAD & Diarization Models from:
- [pyannote audio][https://github.com/pyannote/pyannote-audio]
- [silero vad][https://github.com/snakers4/silero-vad]
- [pyannote-audio](https://github.com/pyannote/pyannote-audio) — Speaker diarization powered by the [speaker-diarization-community-1](https://huggingface.co/pyannote/speaker-diarization-community-1) model, licensed under [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) by [pyannoteAI](https://www.pyannote.ai)
- [silero-vad](https://github.com/snakers4/silero-vad)
Great backend from [faster-whisper](https://github.com/guillaumekln/faster-whisper) and [CTranslate2](https://github.com/OpenNMT/CTranslate2)
+15 -11
View File
@@ -2,25 +2,23 @@
urls = { repository = "https://github.com/m-bain/whisperx" }
authors = [{ name = "Max Bain" }]
name = "whisperx"
version = "3.7.2"
version = "3.8.1"
description = "Time-Accurate Automatic Speech Recognition using Whisper."
readme = "README.md"
requires-python = ">=3.9, <3.14"
requires-python = ">=3.10, <3.14"
license = { text = "BSD-2-Clause" }
dependencies = [
"ctranslate2>=4.5.0",
"faster-whisper>=1.1.1",
"nltk>=3.9.1",
# Restrict numpy, onnxruntime, pandas, av to be compatible with Python 3.9
"numpy>=2.0.2,<2.1.0; python_version <'3.13'",
"onnxruntime>=1.19,<1.20.0; python_version <'3.10'",
"pandas>=2.2.3,<2.3.0",
"av<16.0.0",
"numpy>=2.1.0,<2.3.0; python_version >='3.13'",
"pyannote-audio>=3.3.2,<4.0.0",
"torch>=2.7.1",
"torchaudio",
"numpy>=2.1.0",
"omegaconf>=2.3.0",
"pandas>=2.2.3",
"pyannote-audio>=4.0.0",
"huggingface-hub<1.0.0",
"torch~=2.8.0",
"torchaudio~=2.8.0",
"transformers>=4.48.0",
"triton>=3.3.0; sys_platform == 'linux' and platform_machine == 'x86_64'" # only install triton on x86_64 Linux
]
@@ -39,6 +37,12 @@ include-package-data = true
where = ["."]
include = ["whisperx*"]
# torchcodec (transitive dep of pyannote-audio >=4) has no wheels for Linux aarch64
[tool.uv]
override-dependencies = [
"torchcodec>=0.6.0; (sys_platform == 'linux' and platform_machine == 'x86_64') or sys_platform == 'darwin' or sys_platform == 'win32'",
]
[tool.uv.sources]
torch = [
{ index = "pytorch-cpu", marker = "sys_platform == 'darwin'" },
Generated
+423 -1422
View File
File diff suppressed because it is too large. Load diff
+4 -3
View File
@@ -16,10 +16,10 @@ def cli():
parser.add_argument("--model", default="small", help="name of the Whisper model to use")
parser.add_argument("--model_cache_only", type=str2bool, default=False, help="If True, will not attempt to download models, instead using cached models from --model_dir")
parser.add_argument("--model_dir", type=str, default=None, help="the path to save model files; uses ~/.cache/whisper by default")
parser.add_argument("--device", default="cuda" if torch.cuda.is_available() else "cpu", help="device to use for PyTorch inference")
parser.add_argument("--device", default="cuda" if torch.cuda.is_available() else "cpu", help="device type to use for PyTorch inference (e.g. cpu, cuda)")
parser.add_argument("--device_index", default=0, type=int, help="device index to use for FasterWhisper inference")
parser.add_argument("--batch_size", default=8, type=int, help="the preferred batch size for inference")
parser.add_argument("--compute_type", default="float16", type=str, choices=["float16", "float32", "int8"], help="compute type for computation")
parser.add_argument("--compute_type", default="default", type=str, choices=["default", "float16", "float32", "int8"], help="compute type for computation; 'default' uses float16 on GPU, float32 on CPU")
parser.add_argument("--output_dir", "-o", type=str, default=".", help="directory to save the outputs")
parser.add_argument("--output_format", "-f", type=str, default="all", choices=["all", "srt", "vtt", "txt", "tsv", "json", "aud"], help="format of the output file; if not specified, all available formats will be produced")
@@ -45,7 +45,7 @@ def cli():
parser.add_argument("--diarize", action="store_true", help="Apply diarization to assign speaker labels to each segment/word")
parser.add_argument("--min_speakers", default=None, type=int, help="Minimum number of speakers to in audio file")
parser.add_argument("--max_speakers", default=None, type=int, help="Maximum number of speakers to in audio file")
parser.add_argument("--diarize_model", default="pyannote/speaker-diarization-3.1", type=str, help="Name of the speaker diarization model to use")
parser.add_argument("--diarize_model", default="pyannote/speaker-diarization-community-1", type=str, help="Name of the speaker diarization model to use")
parser.add_argument("--speaker_embeddings", action="store_true", help="Include speaker embeddings in JSON output (only works with --diarize)")
parser.add_argument("--temperature", type=float, default=0, help="temperature to use for sampling")
@@ -58,6 +58,7 @@ def cli():
parser.add_argument("--suppress_numerals", action="store_true", help="whether to suppress numeric symbols and currency symbols during sampling, since wav2vec2 cannot align them correctly")
parser.add_argument("--initial_prompt", type=str, default=None, help="optional text to provide as a prompt for the first window.")
parser.add_argument("--hotwords", type=str, default=None, help="hotwords/hint phrases to the model (e.g. \"WhisperX, PyAnnote, GPU\"); improves recognition of rare/technical terms")
parser.add_argument("--condition_on_previous_text", type=str2bool, default=False, help="if True, provide the previous output of the model as a prompt for the next window; disabling may make the text inconsistent across windows, but the model becomes less prone to getting stuck in a failure loop")
parser.add_argument("--fp16", type=str2bool, default=True, help="whether to perform inference in fp16; True by default")
+20 -8
View File
@@ -14,7 +14,7 @@ import torchaudio
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
from whisperx.audio import SAMPLE_RATE, load_audio
from whisperx.utils import interpolate_nans
from whisperx.utils import interpolate_nans, PUNKT_LANGUAGES
from whisperx.schema import (
AlignedTranscriptionResult,
SingleSegment,
@@ -73,10 +73,11 @@ DEFAULT_ALIGN_MODELS_HF = {
"ka": "xsway/wav2vec2-large-xlsr-georgian",
"lv": "jimregan/wav2vec2-large-xlsr-latvian-cv",
"tl": "Khalsuu/filipino-wav2vec2-l-xls-r-300m-official",
"sv": "KBLab/wav2vec2-large-voxrex-swedish",
}
def load_align_model(language_code: str, device: str, model_name: Optional[str] = None, model_dir=None):
def load_align_model(language_code: str, device: str, model_name: Optional[str] = None, model_dir=None, model_cache_only: bool = False):
if model_name is None:
# use default model
if language_code in DEFAULT_ALIGN_MODELS_TORCH:
@@ -97,8 +98,8 @@ def load_align_model(language_code: str, device: str, model_name: Optional[str]
align_dictionary = {c.lower(): i for i, c in enumerate(labels)}
else:
try:
processor = Wav2Vec2Processor.from_pretrained(model_name, cache_dir=model_dir)
align_model = Wav2Vec2ForCTC.from_pretrained(model_name, cache_dir=model_dir)
processor = Wav2Vec2Processor.from_pretrained(model_name, cache_dir=model_dir, local_files_only=model_cache_only)
align_model = Wav2Vec2ForCTC.from_pretrained(model_name, cache_dir=model_dir, local_files_only=model_cache_only)
except Exception as e:
print(e)
print(f"Error loading model from huggingface, check https://huggingface.co/models for finetuned wav2vec2.0 models")
@@ -191,11 +192,13 @@ def align(
clean_wdx.append(wdx)
# Use language-specific Punkt model if available otherwise we fallback to English.
punkt_lang = PUNKT_LANGUAGES.get(model_lang, 'english')
try:
sentence_splitter = nltk_load('tokenizers/punkt/english.pickle')
sentence_splitter = nltk_load(f'tokenizers/punkt_tab/{punkt_lang}.pickle')
except LookupError:
nltk.download('punkt_tab', quiet=True)
sentence_splitter = nltk_load('tokenizers/punkt/english.pickle')
sentence_splitter = nltk_load(f'tokenizers/punkt_tab/{punkt_lang}.pickle')
sentence_spans = list(sentence_splitter.span_tokenize(text))
segment_data[sdx] = {
@@ -213,6 +216,7 @@ def align(
t1 = segment["start"]
t2 = segment["end"]
text = segment["text"]
avg_logprob = segment.get("avg_logprob")
aligned_seg: SingleAlignedSegment = {
"start": t1,
@@ -222,6 +226,9 @@ def align(
"chars": None,
}
if avg_logprob is not None:
aligned_seg["avg_logprob"] = avg_logprob
if return_char_alignments:
aligned_seg["chars"] = []
@@ -350,12 +357,15 @@ def align(
sentence_words.append(word_segment)
aligned_subsegments.append({
subsegment = {
"text": sentence_text,
"start": sentence_start,
"end": sentence_end,
"words": sentence_words,
})
}
if avg_logprob is not None:
subsegment["avg_logprob"] = avg_logprob
aligned_subsegments.append(subsegment)
if return_char_alignments:
curr_chars = curr_chars[["char", "start", "end", "score"]]
@@ -373,6 +383,8 @@ def align(
agg_dict["text"] = "".join
if return_char_alignments:
agg_dict["chars"] = "sum"
if avg_logprob is not None:
agg_dict["avg_logprob"] = "first"
aligned_subsegments= aligned_subsegments.groupby(["start", "end"], as_index=False).agg(agg_dict)
aligned_subsegments = aligned_subsegments.to_dict('records')
aligned_segments += aligned_subsegments
+27 -12
View File
@@ -58,11 +58,7 @@ class WhisperModel(faster_whisper.WhisperModel):
)
encoder_output = self.encode(features)
max_initial_timestamp_index = int(
round(options.max_initial_timestamp / self.time_precision)
)
result = self.model.generate(
encoder_output,
[prompt] * batch_size,
@@ -72,11 +68,20 @@ class WhisperModel(faster_whisper.WhisperModel):
max_length=self.max_length,
suppress_blank=options.suppress_blank,
suppress_tokens=options.suppress_tokens,
no_repeat_ngram_size=options.no_repeat_ngram_size,
repetition_penalty=options.repetition_penalty,
return_scores=True,
)
tokens_batch = [x.sequences_ids[0] for x in result]
def decode_batch(tokens: List[List[int]]) -> str:
avg_logprobs = []
for res in result:
seq_len = len(res.sequences_ids[0])
cum_logprob = res.scores[0] * (seq_len ** options.length_penalty)
avg_logprobs.append(cum_logprob / (seq_len + 1))
def decode_batch(tokens: List[List[int]]) -> List[str]:
res = []
for tk in tokens:
res.append([token for token in tk if token < tokenizer.eot])
@@ -85,7 +90,7 @@ class WhisperModel(faster_whisper.WhisperModel):
text = decode_batch(tokens_batch)
return text
return {'text': text, 'avg_logprob': avg_logprobs}
def encode(self, features: np.ndarray) -> ctranslate2.StorageView:
# When the model is running on multiple GPUs, the encoder output should be moved
@@ -163,7 +168,7 @@ class FasterWhisperPipeline(Pipeline):
def _forward(self, model_inputs):
outputs = self.model.generate_segment_batched(model_inputs['inputs'], self.tokenizer, self.options)
return {'text': outputs}
return outputs
def postprocess(self, model_outputs):
return model_outputs
@@ -264,15 +269,18 @@ class FasterWhisperPipeline(Pipeline):
percent_complete = base_progress / 2 if combined_progress else base_progress
print(f"Progress: {percent_complete:.2f}%...")
text = out['text']
avg_logprob = out['avg_logprob']
if batch_size in [0, 1, None]:
text = text[0]
avg_logprob = avg_logprob[0]
if verbose:
print(f"Transcript: [{round(vad_segments[idx]['start'], 3)} --> {round(vad_segments[idx]['end'], 3)}] {text}")
segments.append(
{
"text": text,
"start": round(vad_segments[idx]['start'], 3),
"end": round(vad_segments[idx]['end'], 3)
"end": round(vad_segments[idx]['end'], 3),
"avg_logprob": avg_logprob,
}
)
@@ -305,7 +313,7 @@ def load_model(
whisper_arch: str,
device: str,
device_index=0,
compute_type="float16",
compute_type="default",
asr_options: Optional[dict] = None,
language: Optional[str] = None,
vad_model: Optional[Vad]= None,
@@ -316,12 +324,14 @@ def load_model(
download_root: Optional[str] = None,
local_files_only=False,
threads=4,
use_auth_token: Optional[Union[str, bool]] = None,
) -> FasterWhisperPipeline:
"""Load a Whisper model for inference.
Args:
whisper_arch - The name of the Whisper model to load.
device - The device to load the model on.
compute_type - The compute type to use for the model.
Use "default" to automatically select based on device (float16 for GPU, float32 for CPU).
vad_model - The vad model to manually assign.
vad_method - The vad method to use. vad_model has a higher priority if it is not None.
options - A dictionary of options to use for the model.
@@ -334,6 +344,10 @@ def load_model(
A Whisper pipeline.
"""
if compute_type == "default":
compute_type = "float16" if device == "cuda" else "float32"
logger.info(f"Compute type not specified, defaulting to {compute_type} for device {device}")
if whisper_arch.endswith(".en"):
language = "en"
@@ -343,7 +357,8 @@ def load_model(
compute_type=compute_type,
download_root=download_root,
local_files_only=local_files_only,
cpu_threads=threads)
cpu_threads=threads,
use_auth_token=use_auth_token)
if language is not None:
tokenizer = Tokenizer(model.hf_tokenizer, model.model.is_multilingual, task=task, language=language)
else:
@@ -409,7 +424,7 @@ def load_model(
device_vad = f'cuda:{device_index}'
else:
device_vad = device
vad_model = Pyannote(torch.device(device_vad), use_auth_token=None, **default_vad_options)
vad_model = Pyannote(torch.device(device_vad), token=None, **default_vad_options)
else:
raise ValueError(f"Invalid vad_method: {vad_method}")
+143 -47
View File
@@ -1,7 +1,7 @@
import numpy as np
import pandas as pd
from pyannote.audio import Pipeline
from typing import Optional, Union
from typing import Optional, Union, List, Tuple
import torch
from whisperx.audio import load_audio, SAMPLE_RATE
@@ -11,18 +11,96 @@ from whisperx.log_utils import get_logger
logger = get_logger(__name__)
class IntervalTree:
"""
Simple interval tree for fast overlap queries using sorted array + binary search.
Uses O(n) space and provides O(log n) query time instead of O(n) linear scan.
This achieves ~228x speedup for speaker assignment in long-form content.
"""
def __init__(self, intervals: List[Tuple[float, float, str]]):
"""
Initialize the interval tree with diarization segments.
Args:
intervals: List of (start, end, speaker) tuples
"""
if not intervals:
self.starts = np.array([])
self.ends = np.array([])
self.speakers: List[str] = []
return
# Sort intervals by start time for binary search
sorted_intervals = sorted(intervals, key=lambda x: x[0])
self.starts = np.array([i[0] for i in sorted_intervals], dtype=np.float64)
self.ends = np.array([i[1] for i in sorted_intervals], dtype=np.float64)
self.speakers = [i[2] for i in sorted_intervals]
def query(self, start: float, end: float) -> List[Tuple[str, float]]:
"""
Find all intervals that overlap with [start, end] and compute intersection.
Args:
start: Query interval start time
end: Query interval end time
Returns:
List of (speaker, intersection_duration) tuples for overlapping segments
"""
if len(self.starts) == 0:
return []
# Binary search to find candidate intervals
# Only intervals with start < end could overlap
right_idx = np.searchsorted(self.starts, end, side='left')
if right_idx == 0:
return []
# Check candidates for actual overlap
candidates = slice(0, right_idx)
overlaps = (self.starts[candidates] < end) & (self.ends[candidates] > start)
results = []
for idx in np.where(overlaps)[0]:
intersection = min(self.ends[idx], end) - max(self.starts[idx], start)
if intersection > 0:
results.append((self.speakers[idx], intersection))
return results
def find_nearest(self, time: float) -> Optional[str]:
"""
Find the speaker of the nearest segment to a given time point.
Args:
time: Time point to find nearest segment for
Returns:
Speaker ID of nearest segment, or None if no segments exist
"""
if len(self.starts) == 0:
return None
# Calculate midpoints of all segments
mids = (self.starts + self.ends) / 2
nearest_idx = np.argmin(np.abs(mids - time))
return self.speakers[nearest_idx]
class DiarizationPipeline:
def __init__(
self,
model_name=None,
use_auth_token=None,
token=None,
device: Optional[Union[str, torch.device]] = "cpu",
cache_dir=None,
):
if isinstance(device, str):
device = torch.device(device)
model_config = model_name or "pyannote/speaker-diarization-3.1"
model_config = model_name or "pyannote/speaker-diarization-community-1"
logger.info(f"Loading diarization model: {model_config}")
self.model = Pipeline.from_pretrained(model_config, use_auth_token=use_auth_token).to(device)
self.model = Pipeline.from_pretrained(model_config, token=token, cache_dir=cache_dir).to(device)
def __call__(
self,
@@ -55,22 +133,15 @@ class DiarizationPipeline:
'sample_rate': SAMPLE_RATE
}
if return_embeddings:
diarization, embeddings = self.model(
audio_data,
num_speakers=num_speakers,
min_speakers=min_speakers,
max_speakers=max_speakers,
return_embeddings=True,
)
else:
diarization = self.model(
audio_data,
num_speakers=num_speakers,
min_speakers=min_speakers,
max_speakers=max_speakers,
)
embeddings = None
output = self.model(
audio_data,
num_speakers=num_speakers,
min_speakers=min_speakers,
max_speakers=max_speakers,
)
diarization = output.speaker_diarization
embeddings = output.speaker_embeddings if return_embeddings else None
diarize_df = pd.DataFrame(diarization.itertracks(yield_label=True), columns=['segment', 'label', 'speaker'])
diarize_df['start'] = diarize_df['segment'].apply(lambda x: x.start)
@@ -96,6 +167,9 @@ def assign_word_speakers(
"""
Assign speakers to words and segments in the transcript.
Uses an interval tree for O(log n) overlap queries instead of O(n) linear scan,
achieving ~228x speedup for long-form content (3+ hour podcasts).
Args:
diarize_df: Diarization dataframe from DiarizationPipeline
transcript_result: Transcription result to augment with speaker labels
@@ -105,36 +179,58 @@ def assign_word_speakers(
Returns:
Updated transcript_result with speaker assignments and optionally embeddings
"""
transcript_segments = transcript_result["segments"]
transcript_segments = transcript_result.get("segments", [])
if not transcript_segments or diarize_df is None or len(diarize_df) == 0:
return transcript_result
# Build interval tree from diarization segments for O(log n) queries
intervals = [
(row['start'], row['end'], row['speaker'])
for _, row in diarize_df.iterrows()
]
tree = IntervalTree(intervals)
for seg in transcript_segments:
# assign speaker to segment (if any)
diarize_df['intersection'] = np.minimum(diarize_df['end'], seg['end']) - np.maximum(diarize_df['start'], seg['start'])
diarize_df['union'] = np.maximum(diarize_df['end'], seg['end']) - np.minimum(diarize_df['start'], seg['start'])
# remove no hit, otherwise we look for closest (even negative intersection...)
if not fill_nearest:
dia_tmp = diarize_df[diarize_df['intersection'] > 0]
else:
dia_tmp = diarize_df
if len(dia_tmp) > 0:
# sum over speakers
speaker = dia_tmp.groupby("speaker")["intersection"].sum().sort_values(ascending=False).index[0]
seg["speaker"] = speaker
# assign speaker to words
seg_start = seg.get('start', 0.0)
seg_end = seg.get('end', 0.0)
# Query overlapping segments using interval tree
overlaps = tree.query(seg_start, seg_end)
if overlaps:
# Sum intersection durations per speaker and pick the dominant one
speaker_intersections: dict[str, float] = {}
for speaker, intersection in overlaps:
speaker_intersections[speaker] = speaker_intersections.get(speaker, 0.0) + intersection
seg['speaker'] = max(speaker_intersections.items(), key=lambda x: x[1])[0]
elif fill_nearest:
# Find nearest segment if no overlap
seg_mid = (seg_start + seg_end) / 2
nearest_speaker = tree.find_nearest(seg_mid)
if nearest_speaker:
seg['speaker'] = nearest_speaker
# Assign speaker to words
if 'words' in seg:
for word in seg['words']:
if 'start' in word:
diarize_df['intersection'] = np.minimum(diarize_df['end'], word['end']) - np.maximum(diarize_df['start'], word['start'])
diarize_df['union'] = np.maximum(diarize_df['end'], word['end']) - np.minimum(diarize_df['start'], word['start'])
# remove no hit
if not fill_nearest:
dia_tmp = diarize_df[diarize_df['intersection'] > 0]
else:
dia_tmp = diarize_df
if len(dia_tmp) > 0:
# sum over speakers
speaker = dia_tmp.groupby("speaker")["intersection"].sum().sort_values(ascending=False).index[0]
word["speaker"] = speaker
if 'start' not in word:
continue
word_start = word['start']
word_end = word.get('end', word_start)
word_overlaps = tree.query(word_start, word_end)
if word_overlaps:
speaker_intersections = {}
for speaker, intersection in word_overlaps:
speaker_intersections[speaker] = speaker_intersections.get(speaker, 0.0) + intersection
word['speaker'] = max(speaker_intersections.items(), key=lambda x: x[1])[0]
elif fill_nearest:
word_mid = (word_start + word_end) / 2
nearest_speaker = tree.find_nearest(word_mid)
if nearest_speaker:
word['speaker'] = nearest_speaker
# Add speaker embeddings to the result if provided
if speaker_embeddings is not None:
+7
View File
@@ -1,5 +1,10 @@
from typing import TypedDict, Optional, List, Tuple
try:
from typing import NotRequired
except ImportError:
from typing_extensions import NotRequired
class SingleWordSegment(TypedDict):
"""
@@ -28,6 +33,7 @@ class SingleSegment(TypedDict):
start: float
end: float
text: str
avg_logprob: NotRequired[float]
class SegmentData(TypedDict):
@@ -49,6 +55,7 @@ class SingleAlignedSegment(TypedDict):
start: float
end: float
text: str
avg_logprob: NotRequired[float]
words: List[SingleWordSegment]
chars: Optional[List[SingleCharSegment]]
+5 -3
View File
@@ -106,6 +106,7 @@ def transcribe_task(args: dict, parser: argparse.ArgumentParser):
"no_speech_threshold": args.pop("no_speech_threshold"),
"condition_on_previous_text": False,
"initial_prompt": args.pop("initial_prompt"),
"hotwords": args.pop("hotwords"),
"suppress_tokens": [int(x) for x in args.pop("suppress_tokens").split(",")],
"suppress_numerals": args.pop("suppress_numerals"),
}
@@ -140,6 +141,7 @@ def transcribe_task(args: dict, parser: argparse.ArgumentParser):
task=task,
local_files_only=model_cache_only,
threads=faster_whisper_threads,
use_auth_token=hf_token,
)
for audio_path in args.pop("audio"):
@@ -165,7 +167,7 @@ def transcribe_task(args: dict, parser: argparse.ArgumentParser):
tmp_results = results
results = []
align_model, align_metadata = load_align_model(
align_language, device, model_name=align_model
align_language, device, model_name=align_model, model_dir=model_dir, model_cache_only=model_cache_only
)
for result, audio_path in tmp_results:
# >> Align
@@ -182,7 +184,7 @@ def transcribe_task(args: dict, parser: argparse.ArgumentParser):
f"New language found ({result['language']})! Previous was ({align_metadata['language']}), loading new alignment model for new language..."
)
align_model, align_metadata = load_align_model(
result["language"], device
result["language"], device, model_dir=model_dir, model_cache_only=model_cache_only
)
logger.info("Performing alignment...")
result: AlignedTranscriptionResult = align(
@@ -213,7 +215,7 @@ def transcribe_task(args: dict, parser: argparse.ArgumentParser):
logger.info("Performing diarization...")
logger.info(f"Using model: {diarize_model_name}")
results = []
diarize_model = DiarizationPipeline(model_name=diarize_model_name, use_auth_token=hf_token, device=device)
diarize_model = DiarizationPipeline(model_name=diarize_model_name, token=hf_token, device=device, cache_dir=model_dir)
for result, input_audio_path in tmp_results:
diarize_result = diarize_model(
input_audio_path,
+38 -6
View File
@@ -126,6 +126,29 @@ TO_LANGUAGE_CODE = {
LANGUAGES_WITHOUT_SPACES = ["ja", "zh"]
# Mapping of language codes to NLTK Punkt tokenizer model names
PUNKT_LANGUAGES = {
'cs': 'czech',
'da': 'danish',
'de': 'german',
'el': 'greek',
'en': 'english',
'es': 'spanish',
'et': 'estonian',
'fi': 'finnish',
'fr': 'french',
'it': 'italian',
'nl': 'dutch',
'no': 'norwegian',
'pl': 'polish',
'pt': 'portuguese',
'sl': 'slovene',
'sv': 'swedish',
'tr': 'turkish',
"ml": "malayalam",
"ru": "russian",
}
system_encoding = sys.getdefaultencoding()
if system_encoding != "utf-8":
@@ -283,10 +306,19 @@ class SubtitlesWriter(ResultWriter):
yield subtitle, times
if "words" in result["segments"][0]:
for subtitle, _ in iterate_subtitles():
sstart, ssend, speaker = _[0]
subtitle_start = self.format_timestamp(sstart)
subtitle_end = self.format_timestamp(ssend)
for subtitle, times in iterate_subtitles():
speaker = times[0][2]
# Derive cue times from word-level timestamps when available,
# falling back to segment-level times for fully unalignable subtitles.
word_starts = [w["start"] for w in subtitle if "start" in w]
word_ends = [w["end"] for w in subtitle if "end" in w]
if word_starts and word_ends:
subtitle_start = self.format_timestamp(min(word_starts))
subtitle_end = self.format_timestamp(max(word_ends))
else:
subtitle_start = self.format_timestamp(times[0][0])
subtitle_end = self.format_timestamp(times[0][1])
if result["language"] in LANGUAGES_WITHOUT_SPACES:
subtitle_text = "".join([word["word"] for word in subtitle])
else:
@@ -410,7 +442,7 @@ class WriteJSON(ResultWriter):
def get_writer(
output_format: str, output_dir: str
) -> Callable[[dict, TextIO, dict], None]:
) -> Callable[[dict, str, dict], None]:
writers = {
"txt": WriteTXT,
"vtt": WriteVTT,
@@ -425,7 +457,7 @@ def get_writer(
if output_format == "all":
all_writers = [writer(output_dir) for writer in writers.values()]
def write_all(result: dict, file: TextIO, options: dict):
def write_all(result: dict, file: str, options: dict):
for writer in all_writers:
writer(result, file, options)
+6 -6
View File
@@ -18,7 +18,7 @@ from whisperx.log_utils import get_logger
logger = get_logger(__name__)
def load_vad_model(device, vad_onset=0.500, vad_offset=0.363, use_auth_token=None, model_fp=None):
def load_vad_model(device, vad_onset=0.500, vad_offset=0.363, token=None, model_fp=None):
model_dir = torch.hub._get_torch_home()
main_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
@@ -40,7 +40,7 @@ def load_vad_model(device, vad_onset=0.500, vad_offset=0.363, use_auth_token=Non
model_bytes = open(model_fp, "rb").read()
vad_model = Model.from_pretrained(model_fp, use_auth_token=use_auth_token)
vad_model = Model.from_pretrained(model_fp, token=token)
hyperparameters = {"onset": vad_onset,
"offset": vad_offset,
"min_duration_on": 0.1,
@@ -192,11 +192,11 @@ class VoiceActivitySegmentation(VoiceActivityDetection):
self,
segmentation: PipelineModel = "pyannote/segmentation",
fscore: bool = False,
use_auth_token: Union[Text, None] = None,
token: Union[Text, None] = None,
**inference_kwargs,
):
super().__init__(segmentation=segmentation, fscore=fscore, use_auth_token=use_auth_token, **inference_kwargs)
super().__init__(segmentation=segmentation, fscore=fscore, token=token, **inference_kwargs)
def apply(self, file: AudioFile, hook: Optional[Callable] = None) -> Annotation:
"""Apply voice activity detection
@@ -234,10 +234,10 @@ class VoiceActivitySegmentation(VoiceActivityDetection):
class Pyannote(Vad):
def __init__(self, device, use_auth_token=None, model_fp=None, **kwargs):
def __init__(self, device, token=None, model_fp=None, **kwargs):
logger.info("Performing voice activity detection using Pyannote...")
super().__init__(kwargs['vad_onset'])
self.vad_pipeline = load_vad_model(device, use_auth_token=use_auth_token, model_fp=model_fp)
self.vad_pipeline = load_vad_model(device, token=token, model_fp=model_fp)
def __call__(self, audio: AudioFile, **kwargs):
return self.vad_pipeline(audio)