Compare commits

...
51 Commits
Author SHA1 Message Date
Max Bain 064f737998 fix: default compute_type to float32 on CPU to avoid float16 ValueError 2026-02-22 16:12:55 +00:00
Claude-Assistant d8a078eed4 feat: expose avg_logprob per segment from ctranslate2 beam search
Pass through the average log probability (transcription confidence score)
from ctranslate2 to the final segment output. The field is NotRequired
so existing code constructing segments without it remains valid.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-19 07:48:34 +01:00
carlito 4017efc273 Merge pull request #1356 from m-bain/revert-1355-batch_wrap
Revert "Batch wrap"
2026-02-17 13:34:20 +00:00
carlito 0e073d418c Revert "Batch wrap" 2026-02-17 13:33:06 +00:00
carlito 422c44f952 Merge pull request #1355 from 1carlito/batch_wrap
Batch wrap
2026-02-17 13:31:15 +00:00
carlos 1b6a3b78a6 [feat] First batch wrap around
[New File] benchmark testing
2026-02-17 13:30:03 +00:00
carlos de5fa652a1 [modification]
added and condition before streams, existing logic is not chnaged.
2026-02-16 15:43:01 +00:00
carlos e33bb1ec25 Although the existing commit worked, inital prompt was in the loop no need. 2026-02-16 15:43:01 +00:00
carlos 1430e43300 [fix] Batch context is updated each time.
It works with the initial prompt added.

Ran pdb to make sure and check output.

Long audio works.

Existing Logic is correct without flag.
2026-02-16 15:43:01 +00:00
Barabazs 42beab1f9f chore: bump version to 3.8.1 2026-02-14 15:00:56 +01:00
Barabazs 1baf8d2314 feat: pass --hf_token to WhisperModel for gated model support
Forward the existing --hf_token CLI argument to faster-whisper's
WhisperModel via a new use_auth_token parameter on load_model(),
enabling downloads of gated/private HuggingFace models.
2026-02-14 14:48:04 +01:00
MrPrayerandBarabazs 9d687e015e fix: propagate --model_dir and --model_cache_only to all model loading paths (#1285)
- Add `model_cache_only` param to `load_align_model()`, pass as `local_files_only` to HuggingFace `from_pretrained` calls
- Forward `model_dir` and `model_cache_only` to both `load_align_model` call sites (initial load and language-change reload)
- Add `cache_dir` param to `DiarizationPipeline.__init__`, forward to pyannote `Pipeline.from_pretrained`
- Pass `--model_dir` as `cache_dir` when constructing `DiarizationPipeline` in CLI

Previously only the ASR model respected these flags. Alignment and diarization models would always download from HuggingFace to the default cache, breaking offline and custom-cache workflows.


---------

Co-authored-by: Barabazs <31799121+Barabazs@users.noreply.github.com>
2026-02-14 14:11:52 +01:00
BarabazsandGiorgio Azzinnaro 6187d25a44 feat: migrate to pyannote-audio v4 with speaker-diarization-community-1 (#1349)
* feat: upgrade pyannote-audio dependency to v4

* fix: rename use_auth_token to token for pyannote-audio v4 compatibility

* fix: add omegaconf dep

* fix: use structured output API for pyannote-audio v4 diarization

pyannote-audio 4.x no longer returns a plain Annotation (or tuple when>
return_embeddings=True). It now returns a structured output with
speaker_diarization and speaker_embeddings attributes.

* feat: switch default diarization model to speaker-diarization-community-1

Update default from pyannote/speaker-diarization-3.1 to
pyannote/speaker-diarization-community-1 (pyannote-audio v4),
add CC-BY-4.0 attribution, and update README for v4 API changes.

* fix: correct markdown link formatting for silero-vad in README.md

* chore: update version to 3.8.0


Co-authored-by: Giorgio Azzinnaro <giorgio@azzinna.ro>
2026-02-13 13:52:32 -07:00
BarabazsandClaude Opus 4.6 c4c1242edb fix: derive SRT/VTT cue times from word-level timestamps (#1347)
* fix: derive SRT/VTT cue times from word-level timestamps (#1315)

Subtitle cue start/end times were sourced from VAD segment boundaries
instead of word-level timestamps from forced alignment. This caused cues
to appear prematurely and could produce backwards chronological ordering
when VAD segments overlap.

Use min(word starts) / max(word ends) for cue timing, falling back to
segment-level times only when all words are unalignable.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: bump version to 3.7.7 in pyproject.toml

---------

Co-authored-by: Claude-Assistant <noreply@anthropic.com>
2026-02-13 12:53:49 +01:00
carlito 741ab9a2a8 Merge pull request #1343 from m-bain/fix-type-hint-decode-batch
[BugFix] Type hint fix in decode_batch List[str] not str:
2026-02-10 14:46:45 +00:00
carlito f2d853a817 Merge pull request #1342 from 1carlito/bugs
[BugFix] The variable I removed was not being used anyhwere.
2026-02-10 14:39:14 +00:00
carlos cc6f627a39 [BugFix] The variable I removed was not being used anyhwere. 2026-02-10 14:38:13 +00:00
Max Bain 863a986854 Merge pull request #1340 from RickSanchez93/main
Fix: pass no_repeat_ngram_size and repetition_penalty to CTranslate2 generate()
2026-02-10 13:02:21 +00:00
p.sacco 570b08b7be fix: add no_repeat_ngram_size and repetition_penalty options to WhisperModel 2026-02-10 10:44:58 +00:00
carlos 2b5ae8cafa [BugFix] Type hint fix in decode_batch List[str] not str: 2026-02-09 09:55:29 +00:00
Max Bain 66ada295c9 Merge pull request #1338 from Mr-Neutr0n/perf/interval-tree-speaker-assignment
Optimize assign_word_speakers with interval tree for 228x speedup
2026-02-08 18:21:32 +00:00
Mr-Neutr0n 7892a72a30 Optimize assign_word_speakers with interval tree for 228x speedup
Replace O(n*m) pandas operations with O(n log m) interval tree queries
for speaker assignment, where n = words/segments and m = diarization segments.

Performance improvement:
- 7-minute video (1185 words, 147 segments): 73.9s -> 0.32s (228x faster)
- 3-hour podcast: Minutes of processing -> Seconds

Changes:
- Add IntervalTree class using sorted array + binary search
- Refactor assign_word_speakers to use interval tree for overlap queries
- Maintain backward compatibility with same function signature
- Identical output to original implementation

The interval tree uses numpy arrays for efficient storage and binary search
(np.searchsorted) for O(log n) candidate finding, then filters candidates
for actual overlaps.

Fixes #1335
2026-02-07 12:55:26 +05:30
Barabazs 6ec4a02048 chore: drop python 3.9 support (#1328)
* chore: drop python 3.9 support

- Update requires-python to >=3.10
- Remove onnxruntime constraint (only needed for 3.9)
- Simplify numpy (remove version markers and upper bound)
- Remove pandas upper bound (<2.3.0 was for 3.9 compat)
- Remove av direct dependency (transitive via faster-whisper)

* chore(ci): remove python 3.9 from workflows

- Update build-and-release to use Python 3.10
- Remove 3.9 from python-compatibility matrix

* chore: bump version to 3.7.6
2026-01-27 10:41:40 +01:00
Barabazs 9de90e2664 fix: pin huggingface-hub<1.0.0 for pyannote-audio compatibility (#1327)
* fix: pin huggingface-hub<1.0.0 for pyannote-audio compatibility

pyannote-audio uses the deprecated `use_auth_token` parameter which was removed in huggingface-hub v1.0.0

* fix: upgrade yanked dependencies

* chore: update version to 3.7.5
2026-01-27 09:45:05 +01:00
Barabazs d32ec3e301 fix: add missing comma 2025-10-21 09:13:50 -06:00
db317c358b feat: add language-aware sentence tokenization (#1269)
* feat: add language-aware sentence tokenization

* feat: add missing punkt languages

---------

Co-authored-by: pulkit <129310466+p1kit@users.noreply.github.com>
Co-authored-by: Barabazs <31799121+Barabazs@users.noreply.github.com>
2025-10-21 15:57:26 +02:00
JulianFP 6e1d1caaf4 fix: incorrect type annotation in get_writer return value
The audio_path attribute that the __call__ method of the ResultWriter class takes is a str, not TextIO
2025-10-17 09:43:43 -06:00
Barabazs c8f7597345 feat: add hotwords argument to CLI for improved recognition of rare terms 2025-10-17 09:21:56 -06:00
Barabazs 5925e5f8c7 docs: add cuDNN troubleshooting for common issues (#1266)
* docs: add troubleshooting guide for cuDNN loading errors

* docs: add cuDNN version incompatibility troubleshooting
2025-10-16 10:56:51 +02:00
Barabazs 617835dc27 chore: upgrade torch and torchaudio dependencies to 2.8.0 2025-10-16 07:41:45 +00:00
Barabazs 92227e7412 fix: lock down torch and torchaudio versions (#1265)
* fix: update torch and torchaudio dependencies to use compatible version specifiers

* chore: update version to 3.7.3
2025-10-16 08:42:10 +02:00
PaffeandPaffe 0fa81b31f1 feat: add Swedish alignment model (#1110)
---------

Co-authored-by: Paffe <paffe@Sigge.home>
2025-10-15 08:08:18 +02:00
Barabazs 505bd9c0b5 chore: refine triton dependency to restrict installation to x86_64 Linux (#1259)
* chore: refine triton dependency to restrict installation to x86_64 Linux

* bump: update version to 3.7.2
2025-10-12 10:38:29 +02:00
Barabazs 895e5a8493 chore: update numpy dependency constraints for Python 3.13 compatibility (#1258)
* chore: update numpy dependency constraints for Python 3.13 compatibility

* bump: update version to 3.7.1
2025-10-12 10:31:44 +02:00
Barabazs a58ff9cb20 bump: update version to 3.7.0 2025-10-10 07:37:17 +00:00
Barabazs d13171cdde feat: add support for python 3.13 (#1256)
* feat: update Python version requirement to be compatible with 3.13

* feat: add Python 3.13 to compatibility matrix

* feat: update onnxruntime dependency for Python version compatibility

* fix: drop onnxruntime restriction for python >= 3.10
2025-10-10 09:36:24 +02:00
Barabazs c1c08c472f bump: update version to 3.6.0 2025-10-10 06:45:00 +00:00
Barabazs a51ae7a81a feat: add centralized logging to replace ad-hoc print statements (#1254)
* feat: add logging utility functions

* feat: add logging setup and log level argument to CLI

* feat: integrate logging across modules
2025-10-10 08:41:06 +02:00
Barabazs 3b1b9a8c4d refactor: rename types.py to schema.py to avoid stdlib conflict 2025-10-09 14:25:58 +02:00
Tomáš Hnyk 027ec57aee doc: update cpu only example (#1164) 2025-10-09 09:34:54 +02:00
3manifold 64e307cc29 chore: remove redundant variable & improve load_model function documentation (#1197)
* Remove redundant variable

* Improve function documentation
2025-10-09 09:32:02 +02:00
Adrian Wan 2663f2edb5 doc: fix diarize import in example script (#1192) 2025-10-09 09:27:07 +02:00
Barabazs c266ac5459 chore: update version to 3.5.0 2025-10-08 09:24:25 +00:00
Jim ChenandGitHub Copilot 95fecb91c8 build: upgrade PyTorch to 2.7.1 with CUDA 12.8 and multi-platform support
- feat: upgrade PyTorch to 2.7.1 and CUDA 12.8
    * Update README setup to require CUDA toolkit 12.8 instead of 12.4 (Linux and Windows)
    * Bump torch dependency from 2.6.0 to 2.7.1
    * Switch the PyTorch CUDA wheel index from cu124 to cu128


- Revert "docs: add troubleshooting section for libcudnn dependencies in README"
    * The issue of relying on two different versions of CUDNN in this project has been resolved.


- build(pyproject): relax python version and constrain package deps
    * Only download torch from PyTorch; obtain all other packages from PyPI.
    * Restrict numpy, onnxruntime, pandas to be compatible with Python 3.9


- build(pyproject): require triton 3.3.0+ for arm64 support
    * Add triton version 3.3.0 or newer to the dependencies to support arm64 architecture.

- build: skip Triton on Windows since it isn't supported
    * Add a platform marker to the triton dependency to skip it on Windows, as triton does not support Windows.

- build: configure PyTorch sources for cross-platform compatibility
    * macOS uses CPU-only PyTorch from pytorch-cpu index
    * Linux and Windows use CUDA 12.8 PyTorch from pytorch index
    * triton only installs on Linux with CUDA 12.8 support
    * Update lockfile to support multi-platform builds

- fix: restrict av to <16.0.0 for Python 3.9 compatibility
    * Add av<16.0.0 to dependencies to maintain Python 3.9 support
    * Update comment to include av in the restriction list
    * Update uv.lock accordingly

        PyAV dropped Python 3.9 support in v16.0.0:
        https://github.com/PyAV-Org/PyAV/commit/106089447cab9dc6d3ca2d1ab7e0aeff76d7a6a4


- fix: resolve PyTorch ARM64 platform compatibility issue

    * Update uv.lock to properly handle aarch64 platforms for PyTorch dependencies
    * Add resolution markers for ARM64 Linux systems to use CPU-only PyTorch builds
    * Ensure CUDA builds are only used on x86_64 platforms where supported

    Resolves ARM64 Docker build failures by preventing uv from attempting to install CUDA PyTorch on unsupported platforms

- chore: change .python-version to 3.10

---

Signed-off-by: CHEN, CHUN <jim60105@gmail.com>
Signed-off-by: Jim Chen <Jim@ChenJ.im>
Co-authored-by: GitHub Copilot <bot@ChenJ.im>
2025-10-08 11:21:28 +02:00
Nguyen Binh b1c8ac7de6 Change alignment model for Vietnamese language
Since the current model is a wav2vec2 pre-trained model for Vietnamese audio, it won't work with alignment tasks. To make it work as expected, I recommend chaining to a fine-tuned ASR version.
2025-10-03 09:41:03 +02:00
Barabazs bf150e442e feat: update Punkt tokenizer to use pre-trained model and handle missing data 2025-10-03 09:11:24 +02:00
Max Bain ed13dc8c6c recall.ai sponsor
Added information about Recall.ai's Meeting Transcription API.
2025-10-03 00:12:53 +01:00
Alex Cannan c7d31883bc Add jr, sr, and ph.d to punkt abbreviations 2025-10-01 08:59:53 +02:00
Barabazs 83afb81ac7 fix: restrict pyannote-audio version to avoid compatibility issues (#1242)
* fix: restrict pyannote-audio version to avoid compatibility issues
* chore: bump whisperX version to 3.4.3
2025-10-01 08:37:00 +02:00
2d9ce44329 fix(asr): load VAD model on correct CUDA device (#835)
fix(asr): load VAD model on correct CUDA device

Previously, the VAD sub‐model was always initialized on the default CUDA device (cuda:0), even when a higher device_index was specified. This change sets `device_vad` to `cuda:{device_index}` whenever `device == 'cuda'`, while falling back to the original `device` string for non‐CUDA cases. This ensures the VAD model is loaded on the intended GPU.


Co-authored-by: dujing <dujing@xmov.ai>
Co-authored-by: Barabazs <31799121+Barabazs@users.noreply.github.com>
2025-07-02 08:07:59 +02:00
3manifold f4261f34e9 Remove unused code in Vad class 2025-07-01 09:06:04 +02:00
19 changed files with 2710 additions and 2062 deletions

No files matched your search

+1 -1
View File
@@ -15,7 +15,7 @@ jobs:
uses: astral-sh/setup-uv@v5
with:
version: "0.5.14"
python-version: "3.9"
python-version: "3.10"
- name: Check if lockfile is up to date
run: uv lock --check
+1 -1
View File
@@ -12,7 +12,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.9", "3.10", "3.11", "3.12"]
python-version: ["3.10", "3.11", "3.12", "3.13"]
steps:
- uses: actions/checkout@v4
+1
View File
@@ -0,0 +1 @@
3.10
+76
View File
@@ -0,0 +1,76 @@
# Troubleshooting cuDNN Loading Errors
This guide helps resolve common cuDNN-related errors when running WhisperX on GPU. These issues typically occur when the system can't locate cuDNN libraries or finds conflicting versions.
## Unable to Load cuDNN Libraries
If you encounter the following error when running WhisperX:
`Unable to load any of {libcudnn_cnn.so.9.1.0, libcudnn_cnn.so.9.1, libcudnn_cnn.so.9, libcudnn_cnn.so}`
This means the cuDNN libraries are installed (via whisperx dependencies) but aren't in a location where the system's dynamic linker can find them.
### Solution 1: Add to LD_LIBRARY_PATH (Recommended)
Add this at the start of your Python script or notebook:
```python
import os
# Get current LD_LIBRARY_PATH
original = os.environ.get("LD_LIBRARY_PATH", "")
cudnn_path = "/usr/local/lib/python3.12/dist-packages/nvidia/cudnn/lib/"
os.environ['LD_LIBRARY_PATH'] = original + ":" + cudnn_path
```
**Note:** Adjust the Python version (`python3.12`) to match your environment.
### Solution 2: Symlink to LD_LIBRARY_PATH Directory
If Solution 1 didn't work and you still get the "unable to load" error, symlink the libraries to a directory that's already in your `LD_LIBRARY_PATH`:
1. Check what's in your LD_LIBRARY_PATH: `echo "$LD_LIBRARY_PATH"`
2. Assuming that there is only one path set.
Symlink the downloaded libcudnn files to that path:
`ln -s /usr/local/lib/python3.12/dist-packages/nvidia/cudnn/lib/libcudnn* "$LD_LIBRARY_PATH"/`
**Note:** If `LD_LIBRARY_PATH` contains multiple paths (separated by `:`), pick one directory and use it instead of `"$LD_LIBRARY_PATH"`. For example: `/usr/lib/x86_64-linux-gnu/`
## cuDNN Version Incompatibility
If you encounter this error:
```
RuntimeError: cuDNN version incompatibility: PyTorch was compiled against (9, 10, 2) but found runtime version (9, 2, 1)
```
This means PyTorch is finding a different cuDNN version than the one it was compiled with. **PyTorch comes bundled with its own cuDNN**, but a conflicting cuDNN in `LD_LIBRARY_PATH` is taking precedence.
### Solution: Remove Conflicting cuDNN from Path
Check if there's a conflicting cuDNN path:
```bash
echo $LD_LIBRARY_PATH
```
If you see paths pointing to older cuDNN installations (e.g., system-installed cuDNN or manually downloaded), try one of these:
**Option 1: Clear LD_LIBRARY_PATH temporarily**
```python
import os
# Let PyTorch use its bundled cuDNN
os.environ.pop('LD_LIBRARY_PATH', None)
```
**Option 2: Set LD_LIBRARY_PATH to only the correct version**
```python
import os
# Point only to the cuDNN that matches PyTorch's compiled version
os.environ['LD_LIBRARY_PATH'] = "/usr/local/lib/python3.12/dist-packages/nvidia/cudnn/lib/"
```
**Note:** This error is unlikely on a clean install. If it occurs anyway, [open an issue](https://github.com/m-bain/whisperX/issues). If you've modified system libraries or CUDA/cuDNN, the options above should help resolve most cases.
+20 -27
View File
@@ -1,5 +1,10 @@
<h1 align="center">WhisperX</h1>
## Recall.ai - Meeting Transcription API
If you’re looking for a transcription API for meetings, consider checking out [Recall.ai's Meeting Transcription API](https://www.recall.ai/product/meeting-transcription-api?utm_source=github&utm_medium=sponsorship&utm_campaign=mbain-whisperx), an API that works with Zoom, Google Meet, Microsoft Teams, and more. Recall.ai diarizes by pulling the speaker data and separate audio streams from the meeting platforms, which means 100% accurate speaker diarization with actual speaker names.
<p align="center">
<a href="https://github.com/m-bain/whisperX/stargazers">
<img src="https://img.shields.io/github/stars/m-bain/whisperX.svg?colorA=orange&colorB=orange&logo=github"
@@ -57,6 +62,15 @@ This repository provides fast automatic speech recognition (70x realtime with la
<h2 align="left" id="setup">Setup ⚙️</h2>
### 0. CUDA Installation
To use WhisperX with GPU acceleration, install the CUDA toolkit 12.8 before WhisperX. Skip this step if using only the CPU.
- For **Linux** users, install the CUDA toolkit 12.8 following this guide:
[CUDA Installation Guide for Linux](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/).
- For **Windows** users, download and install the CUDA toolkit 12.8:
[CUDA Downloads](https://developer.nvidia.com/cuda-12-8-1-download-archive).
### 1. Simple Installation (Recommended)
The easiest way to install WhisperX is through PyPi:
@@ -97,31 +111,9 @@ uv sync --all-extras --dev
You may also need to install ffmpeg, rust etc. Follow openAI instructions here https://github.com/openai/whisper#setup.
### Common Issues & Troubleshooting 🔧
#### libcudnn Dependencies (GPU Users)
If you're using WhisperX with GPU support and encounter errors like:
- `Could not load library libcudnn_ops_infer.so.8`
- `Unable to load any of {libcudnn_cnn.so.9.1.0, libcudnn_cnn.so.9.1, libcudnn_cnn.so.9, libcudnn_cnn.so}`
- `libcudnn_ops_infer.so.8: cannot open shared object file: No such file or directory`
This means your system is missing the CUDA Deep Neural Network library (cuDNN). This library is needed for GPU acceleration but isn't always installed by default.
**Install cuDNN (example for apt based systems):**
```bash
sudo apt update
sudo apt install libcudnn8 libcudnn8-dev -y
```
### Speaker Diarization
To **enable Speaker Diarization**, include your Hugging Face access token (read) that you can generate from [Here](https://huggingface.co/settings/tokens) after the `--hf_token` argument and accept the user agreement for the following models: [Segmentation](https://huggingface.co/pyannote/segmentation-3.0) and [Speaker-Diarization-3.1](https://huggingface.co/pyannote/speaker-diarization-3.1) (if you choose to use Speaker-Diarization 2.x, follow requirements [here](https://huggingface.co/pyannote/speaker-diarization) instead.)
> **Note**<br>
> As of Oct 11, 2023, there is a known issue regarding slow performance with pyannote/Speaker-Diarization-3.0 in whisperX. It is due to dependency conflicts between faster-whisper and pyannote-audio 3.0.0. Please see [this issue](https://github.com/m-bain/whisperX/issues/499) for more details and potential workarounds.
To **enable Speaker Diarization**, include your Hugging Face access token (read) that you can generate from [Here](https://huggingface.co/settings/tokens) after the `--hf_token` argument and accept the user agreement for the [speaker-diarization-community-1](https://huggingface.co/pyannote/speaker-diarization-community-1) model.
<h2 align="left" id="example">Usage 💬 (command line)</h2>
@@ -149,7 +141,7 @@ To label the transcript with speaker ID's (set number of speakers if known e.g.
To run on CPU instead of GPU (and for running on Mac OS X):
whisperx path/to/audio.wav --compute_type int8
whisperx path/to/audio.wav --compute_type int8 --device cpu
### Other languages
@@ -171,6 +163,7 @@ See more examples in other languages [here](EXAMPLES.md).
```python
import whisperx
import gc
from whisperx.diarize import DiarizationPipeline
device = "cuda"
audio_file = "audio.mp3"
@@ -201,7 +194,7 @@ print(result["segments"]) # after alignment
# import gc; import torch; gc.collect(); torch.cuda.empty_cache(); del model_a
# 3. Assign speaker labels
diarize_model = whisperx.diarize.DiarizationPipeline(use_auth_token=YOUR_HF_TOKEN, device=device)
diarize_model = DiarizationPipeline(token=YOUR_HF_TOKEN, device=device)
# add min/max number of speakers if known
diarize_segments = diarize_model(audio)
@@ -295,8 +288,8 @@ And uses the wonderful pyannote VAD / Diarization https://github.com/pyannote/py
Valuable VAD & Diarization Models from:
- [pyannote audio][https://github.com/pyannote/pyannote-audio]
- [silero vad][https://github.com/snakers4/silero-vad]
- [pyannote-audio](https://github.com/pyannote/pyannote-audio) — Speaker diarization powered by the [speaker-diarization-community-1](https://huggingface.co/pyannote/speaker-diarization-community-1) model, licensed under [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) by [pyannoteAI](https://www.pyannote.ai)
- [silero-vad](https://github.com/snakers4/silero-vad)
Great backend from [faster-whisper](https://github.com/guillaumekln/faster-whisper) and [CTranslate2](https://github.com/OpenNMT/CTranslate2)
+41 -8
View File
@@ -2,23 +2,25 @@
urls = { repository = "https://github.com/m-bain/whisperx" }
authors = [{ name = "Max Bain" }]
name = "whisperx"
version = "3.4.2"
version = "3.8.1"
description = "Time-Accurate Automatic Speech Recognition using Whisper."
readme = "README.md"
requires-python = ">=3.9, <3.13"
requires-python = ">=3.10, <3.14"
license = { text = "BSD-2-Clause" }
dependencies = [
"ctranslate2<4.5.0",
"ctranslate2>=4.5.0",
"faster-whisper>=1.1.1",
"nltk>=3.9.1",
"numpy>=2.0.2",
"onnxruntime>=1.19",
"numpy>=2.1.0",
"omegaconf>=2.3.0",
"pandas>=2.2.3",
"pyannote-audio>=3.3.2",
"torch>=2.5.1",
"torchaudio>=2.5.1",
"pyannote-audio>=4.0.0",
"huggingface-hub<1.0.0",
"torch~=2.8.0",
"torchaudio~=2.8.0",
"transformers>=4.48.0",
"triton>=3.3.0; sys_platform == 'linux' and platform_machine == 'x86_64'" # only install triton on x86_64 Linux
]
@@ -34,3 +36,34 @@ include-package-data = true
[tool.setuptools.packages.find]
where = ["."]
include = ["whisperx*"]
# torchcodec (transitive dep of pyannote-audio >=4) has no wheels for Linux aarch64
[tool.uv]
override-dependencies = [
"torchcodec>=0.6.0; (sys_platform == 'linux' and platform_machine == 'x86_64') or sys_platform == 'darwin' or sys_platform == 'win32'",
]
[tool.uv.sources]
torch = [
{ index = "pytorch-cpu", marker = "sys_platform == 'darwin'" },
{ index = "pytorch-cpu", marker = "platform_machine != 'x86_64' and sys_platform != 'darwin'" },
{ index = "pytorch", marker = "platform_machine == 'x86_64' and sys_platform != 'darwin'" },
]
torchaudio = [
{ index = "pytorch-cpu", marker = "sys_platform == 'darwin'" },
{ index = "pytorch-cpu", marker = "platform_machine != 'x86_64' and sys_platform != 'darwin'" },
{ index = "pytorch", marker = "platform_machine == 'x86_64' and sys_platform != 'darwin'" },
]
triton = [
{ index = "pytorch", marker = "sys_platform == 'linux'" },
]
[[tool.uv.index]]
name = "pytorch"
url = "https://download.pytorch.org/whl/cu128"
explicit = true
[[tool.uv.index]]
name = "pytorch-cpu"
url = "https://download.pytorch.org/whl/cpu"
explicit = true
Generated
+2148 -1878
View File
File diff suppressed because it is too large. Load diff
+26
View File
@@ -29,3 +29,29 @@ def load_audio(*args, **kwargs):
def assign_word_speakers(*args, **kwargs):
diarize = _lazy_import("diarize")
return diarize.assign_word_speakers(*args, **kwargs)
def setup_logging(*args, **kwargs):
"""
Configure logging for WhisperX.
Args:
level: Logging level (debug, info, warning, error, critical). Default: warning
log_file: Optional path to log file. If None, logs only to console.
"""
logging_module = _lazy_import("log_utils")
return logging_module.setup_logging(*args, **kwargs)
def get_logger(*args, **kwargs):
"""
Get a logger instance for the given module.
Args:
name: Logger name (typically __name__ from calling module)
Returns:
Logger instance configured with WhisperX settings
"""
logging_module = _lazy_import("log_utils")
return logging_module.get_logger(*args, **kwargs)
+16 -3
View File
@@ -6,6 +6,7 @@ import torch
from whisperx.utils import (LANGUAGES, TO_LANGUAGE_CODE, optional_float,
optional_int, str2bool)
from whisperx.log_utils import setup_logging
def cli():
@@ -15,14 +16,15 @@ def cli():
parser.add_argument("--model", default="small", help="name of the Whisper model to use")
parser.add_argument("--model_cache_only", type=str2bool, default=False, help="If True, will not attempt to download models, instead using cached models from --model_dir")
parser.add_argument("--model_dir", type=str, default=None, help="the path to save model files; uses ~/.cache/whisper by default")
parser.add_argument("--device", default="cuda" if torch.cuda.is_available() else "cpu", help="device to use for PyTorch inference")
parser.add_argument("--device", default="cuda" if torch.cuda.is_available() else "cpu", help="device type to use for PyTorch inference (e.g. cpu, cuda)")
parser.add_argument("--device_index", default=0, type=int, help="device index to use for FasterWhisper inference")
parser.add_argument("--batch_size", default=8, type=int, help="the preferred batch size for inference")
parser.add_argument("--compute_type", default="float16", type=str, choices=["float16", "float32", "int8"], help="compute type for computation")
parser.add_argument("--compute_type", default="default", type=str, choices=["default", "float16", "float32", "int8"], help="compute type for computation; 'default' uses float16 on GPU, float32 on CPU")
parser.add_argument("--output_dir", "-o", type=str, default=".", help="directory to save the outputs")
parser.add_argument("--output_format", "-f", type=str, default="all", choices=["all", "srt", "vtt", "txt", "tsv", "json", "aud"], help="format of the output file; if not specified, all available formats will be produced")
parser.add_argument("--verbose", type=str2bool, default=True, help="whether to print out the progress and debug messages")
parser.add_argument("--log-level", type=str, default=None, choices=["debug", "info", "warning", "error", "critical"], help="logging level (overrides --verbose if set)")
parser.add_argument("--task", type=str, default="transcribe", choices=["transcribe", "translate"], help="whether to perform X->X speech recognition ('transcribe') or X->English translation ('translate')")
parser.add_argument("--language", type=str, default=None, choices=sorted(LANGUAGES.keys()) + sorted([k.title() for k in TO_LANGUAGE_CODE.keys()]), help="language spoken in the audio, specify None to perform language detection")
@@ -43,7 +45,7 @@ def cli():
parser.add_argument("--diarize", action="store_true", help="Apply diarization to assign speaker labels to each segment/word")
parser.add_argument("--min_speakers", default=None, type=int, help="Minimum number of speakers to in audio file")
parser.add_argument("--max_speakers", default=None, type=int, help="Maximum number of speakers to in audio file")
parser.add_argument("--diarize_model", default="pyannote/speaker-diarization-3.1", type=str, help="Name of the speaker diarization model to use")
parser.add_argument("--diarize_model", default="pyannote/speaker-diarization-community-1", type=str, help="Name of the speaker diarization model to use")
parser.add_argument("--speaker_embeddings", action="store_true", help="Include speaker embeddings in JSON output (only works with --diarize)")
parser.add_argument("--temperature", type=float, default=0, help="temperature to use for sampling")
@@ -56,6 +58,7 @@ def cli():
parser.add_argument("--suppress_numerals", action="store_true", help="whether to suppress numeric symbols and currency symbols during sampling, since wav2vec2 cannot align them correctly")
parser.add_argument("--initial_prompt", type=str, default=None, help="optional text to provide as a prompt for the first window.")
parser.add_argument("--hotwords", type=str, default=None, help="hotwords/hint phrases to the model (e.g. \"WhisperX, PyAnnote, GPU\"); improves recognition of rare/technical terms")
parser.add_argument("--condition_on_previous_text", type=str2bool, default=False, help="if True, provide the previous output of the model as a prompt for the next window; disabling may make the text inconsistent across windows, but the model becomes less prone to getting stuck in a failure loop")
parser.add_argument("--fp16", type=str2bool, default=True, help="whether to perform inference in fp16; True by default")
@@ -80,6 +83,16 @@ def cli():
args = parser.parse_args().__dict__
log_level = args.get("log_level")
verbose = args.get("verbose")
if log_level is not None:
setup_logging(level=log_level)
elif verbose:
setup_logging(level="info")
else:
setup_logging(level="warning")
from whisperx.transcribe import transcribe_task
transcribe_task(args, parser)
+47 -30
View File
@@ -14,17 +14,19 @@ import torchaudio
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
from whisperx.audio import SAMPLE_RATE, load_audio
from whisperx.utils import interpolate_nans
from whisperx.types import (
from whisperx.utils import interpolate_nans, PUNKT_LANGUAGES
from whisperx.schema import (
AlignedTranscriptionResult,
SingleSegment,
SingleAlignedSegment,
SingleWordSegment,
SegmentData,
)
from nltk.tokenize.punkt import PunktSentenceTokenizer, PunktParameters
import nltk
from nltk.data import load as nltk_load
from whisperx.log_utils import get_logger
PUNKT_ABBREVIATIONS = ['dr', 'vs', 'mr', 'mrs', 'prof']
logger = get_logger(__name__)
LANGUAGES_WITHOUT_SPACES = ["ja", "zh"]
@@ -53,7 +55,7 @@ DEFAULT_ALIGN_MODELS_HF = {
"tr": "mpoyraz/wav2vec2-xls-r-300m-cv7-turkish",
"da": "saattrupdan/wav2vec2-xls-r-300m-ftspeech",
"he": "imvladikon/wav2vec2-xls-r-300m-hebrew",
"vi": 'nguyenvulebinh/wav2vec2-base-vi',
"vi": 'nguyenvulebinh/wav2vec2-base-vi-vlsp2020',
"ko": "kresnik/wav2vec2-large-xlsr-korean",
"ur": "kingabzpro/wav2vec2-large-xls-r-300m-Urdu",
"te": "anuragshas/wav2vec2-large-xlsr-53-telugu",
@@ -71,10 +73,11 @@ DEFAULT_ALIGN_MODELS_HF = {
"ka": "xsway/wav2vec2-large-xlsr-georgian",
"lv": "jimregan/wav2vec2-large-xlsr-latvian-cv",
"tl": "Khalsuu/filipino-wav2vec2-l-xls-r-300m-official",
"sv": "KBLab/wav2vec2-large-voxrex-swedish",
}
def load_align_model(language_code: str, device: str, model_name: Optional[str] = None, model_dir=None):
def load_align_model(language_code: str, device: str, model_name: Optional[str] = None, model_dir=None, model_cache_only: bool = False):
if model_name is None:
# use default model
if language_code in DEFAULT_ALIGN_MODELS_TORCH:
@@ -82,8 +85,9 @@ def load_align_model(language_code: str, device: str, model_name: Optional[str]
elif language_code in DEFAULT_ALIGN_MODELS_HF:
model_name = DEFAULT_ALIGN_MODELS_HF[language_code]
else:
print(f"There is no default alignment model set for this language ({language_code}).\
Please find a wav2vec2.0 model finetuned on this language in https://huggingface.co/models, then pass the model name in --align_model [MODEL_NAME]")
logger.error(f"No default alignment model for language: {language_code}. "
f"Please find a wav2vec2.0 model finetuned on this language at https://huggingface.co/models, "
f"then pass the model name via --align_model [MODEL_NAME]")
raise ValueError(f"No default align-model for language: {language_code}")
if model_name in torchaudio.pipelines.__all__:
@@ -94,8 +98,8 @@ def load_align_model(language_code: str, device: str, model_name: Optional[str]
align_dictionary = {c.lower(): i for i, c in enumerate(labels)}
else:
try:
processor = Wav2Vec2Processor.from_pretrained(model_name, cache_dir=model_dir)
align_model = Wav2Vec2ForCTC.from_pretrained(model_name, cache_dir=model_dir)
processor = Wav2Vec2Processor.from_pretrained(model_name, cache_dir=model_dir, local_files_only=model_cache_only)
align_model = Wav2Vec2ForCTC.from_pretrained(model_name, cache_dir=model_dir, local_files_only=model_cache_only)
except Exception as e:
print(e)
print(f"Error loading model from huggingface, check https://huggingface.co/models for finetuned wav2vec2.0 models")
@@ -124,14 +128,14 @@ def align(
"""
Align phoneme recognition predictions to known transcription.
"""
if not torch.is_tensor(audio):
if isinstance(audio, str):
audio = load_audio(audio)
audio = torch.from_numpy(audio)
if len(audio.shape) == 1:
audio = audio.unsqueeze(0)
MAX_DURATION = audio.shape[1] / SAMPLE_RATE
model_dictionary = align_model_metadata["dictionary"]
@@ -148,7 +152,7 @@ def align(
base_progress = ((sdx + 1) / total_segments) * 100
percent_complete = (50 + base_progress / 2) if combined_progress else base_progress
print(f"Progress: {percent_complete:.2f}%...")
num_leading = len(segment["text"]) - len(segment["text"].lstrip())
num_trailing = len(segment["text"]) - len(segment["text"].rstrip())
text = segment["text"]
@@ -165,7 +169,7 @@ def align(
# wav2vec2 models use "|" character to represent spaces
if model_lang not in LANGUAGES_WITHOUT_SPACES:
char_ = char_.replace(" ", "|")
# ignore whitespace at beginning and end of transcript
if cdx < num_leading:
pass
@@ -187,10 +191,14 @@ def align(
# index for placeholder
clean_wdx.append(wdx)
punkt_param = PunktParameters()
punkt_param.abbrev_types = set(PUNKT_ABBREVIATIONS)
sentence_splitter = PunktSentenceTokenizer(punkt_param)
# Use language-specific Punkt model if available otherwise we fallback to English.
punkt_lang = PUNKT_LANGUAGES.get(model_lang, 'english')
try:
sentence_splitter = nltk_load(f'tokenizers/punkt_tab/{punkt_lang}.pickle')
except LookupError:
nltk.download('punkt_tab', quiet=True)
sentence_splitter = nltk_load(f'tokenizers/punkt_tab/{punkt_lang}.pickle')
sentence_spans = list(sentence_splitter.span_tokenize(text))
segment_data[sdx] = {
@@ -199,15 +207,16 @@ def align(
"clean_wdx": clean_wdx,
"sentence_spans": sentence_spans
}
aligned_segments: List[SingleAlignedSegment] = []
# 2. Get prediction matrix from alignment model & align
for sdx, segment in enumerate(transcript):
t1 = segment["start"]
t2 = segment["end"]
text = segment["text"]
avg_logprob = segment.get("avg_logprob")
aligned_seg: SingleAlignedSegment = {
"start": t1,
@@ -217,17 +226,20 @@ def align(
"chars": None,
}
if avg_logprob is not None:
aligned_seg["avg_logprob"] = avg_logprob
if return_char_alignments:
aligned_seg["chars"] = []
# check we can align
if len(segment_data[sdx]["clean_char"]) == 0:
print(f'Failed to align segment ("{segment["text"]}"): no characters in this segment found in model dictionary, resorting to original...')
logger.warning(f'Failed to align segment ("{segment["text"]}"): no characters in this segment found in model dictionary, resorting to original')
aligned_segments.append(aligned_seg)
continue
if t1 >= MAX_DURATION:
print(f'Failed to align segment ("{segment["text"]}"): original start time longer than audio duration, skipping...')
logger.warning(f'Failed to align segment ("{segment["text"]}"): original start time longer than audio duration, skipping')
aligned_segments.append(aligned_seg)
continue
@@ -247,7 +259,7 @@ def align(
)
else:
lengths = None
with torch.inference_mode():
if model_type == "torchaudio":
emissions, _ = model(waveform_segment.to(device), lengths=lengths)
@@ -269,7 +281,7 @@ def align(
path = backtrack_beam(trellis, emission, tokens, blank_id, beam_width=2)
if path is None:
print(f'Failed to align segment ("{segment["text"]}"): backtrack failed, resorting to original...')
logger.warning(f'Failed to align segment ("{segment["text"]}"): backtrack failed, resorting to original')
aligned_segments.append(aligned_seg)
continue
@@ -304,7 +316,7 @@ def align(
word_idx += 1
elif cdx == len(text) - 1 or text[cdx+1] == " ":
word_idx += 1
char_segments_arr = pd.DataFrame(char_segments_arr)
aligned_subsegments = []
@@ -333,7 +345,7 @@ def align(
word_end = word_chars["end"].max()
word_score = round(word_chars["score"].mean(), 3)
# -1 indicates unalignable
# -1 indicates unalignable
word_segment = {"word": word_text}
if not np.isnan(word_start):
@@ -344,13 +356,16 @@ def align(
word_segment["score"] = word_score
sentence_words.append(word_segment)
aligned_subsegments.append({
subsegment = {
"text": sentence_text,
"start": sentence_start,
"end": sentence_end,
"words": sentence_words,
})
}
if avg_logprob is not None:
subsegment["avg_logprob"] = avg_logprob
aligned_subsegments.append(subsegment)
if return_char_alignments:
curr_chars = curr_chars[["char", "start", "end", "score"]]
@@ -368,6 +383,8 @@ def align(
agg_dict["text"] = "".join
if return_char_alignments:
agg_dict["chars"] = "sum"
if avg_logprob is not None:
agg_dict["avg_logprob"] = "first"
aligned_subsegments= aligned_subsegments.groupby(["start", "end"], as_index=False).agg(agg_dict)
aligned_subsegments = aligned_subsegments.to_dict('records')
aligned_segments += aligned_subsegments
+41 -18
View File
@@ -12,8 +12,11 @@ from transformers import Pipeline
from transformers.pipelines.pt_utils import PipelineIterator
from whisperx.audio import N_SAMPLES, SAMPLE_RATE, load_audio, log_mel_spectrogram
from whisperx.types import SingleSegment, TranscriptionResult
from whisperx.schema import SingleSegment, TranscriptionResult
from whisperx.vads import Vad, Silero, Pyannote
from whisperx.log_utils import get_logger
logger = get_logger(__name__)
def find_numeral_symbol_tokens(tokenizer):
@@ -55,11 +58,7 @@ class WhisperModel(faster_whisper.WhisperModel):
)
encoder_output = self.encode(features)
max_initial_timestamp_index = int(
round(options.max_initial_timestamp / self.time_precision)
)
result = self.model.generate(
encoder_output,
[prompt] * batch_size,
@@ -69,11 +68,20 @@ class WhisperModel(faster_whisper.WhisperModel):
max_length=self.max_length,
suppress_blank=options.suppress_blank,
suppress_tokens=options.suppress_tokens,
no_repeat_ngram_size=options.no_repeat_ngram_size,
repetition_penalty=options.repetition_penalty,
return_scores=True,
)
tokens_batch = [x.sequences_ids[0] for x in result]
def decode_batch(tokens: List[List[int]]) -> str:
avg_logprobs = []
for res in result:
seq_len = len(res.sequences_ids[0])
cum_logprob = res.scores[0] * (seq_len ** options.length_penalty)
avg_logprobs.append(cum_logprob / (seq_len + 1))
def decode_batch(tokens: List[List[int]]) -> List[str]:
res = []
for tk in tokens:
res.append([token for token in tk if token < tokenizer.eot])
@@ -82,7 +90,7 @@ class WhisperModel(faster_whisper.WhisperModel):
text = decode_batch(tokens_batch)
return text
return {'text': text, 'avg_logprob': avg_logprobs}
def encode(self, features: np.ndarray) -> ctranslate2.StorageView:
# When the model is running on multiple GPUs, the encoder output should be moved
@@ -160,7 +168,7 @@ class FasterWhisperPipeline(Pipeline):
def _forward(self, model_inputs):
outputs = self.model.generate_segment_batched(model_inputs['inputs'], self.tokenizer, self.options)
return {'text': outputs}
return outputs
def postprocess(self, model_outputs):
return model_outputs
@@ -247,7 +255,7 @@ class FasterWhisperPipeline(Pipeline):
if self.suppress_numerals:
previous_suppress_tokens = self.options.suppress_tokens
numeral_symbol_tokens = find_numeral_symbol_tokens(self.tokenizer)
print(f"Suppressing numeral and symbol tokens")
logger.info("Suppressing numeral and symbol tokens")
new_suppressed_tokens = numeral_symbol_tokens + self.options.suppress_tokens
new_suppressed_tokens = list(set(new_suppressed_tokens))
self.options = replace(self.options, suppress_tokens=new_suppressed_tokens)
@@ -261,15 +269,18 @@ class FasterWhisperPipeline(Pipeline):
percent_complete = base_progress / 2 if combined_progress else base_progress
print(f"Progress: {percent_complete:.2f}%...")
text = out['text']
avg_logprob = out['avg_logprob']
if batch_size in [0, 1, None]:
text = text[0]
avg_logprob = avg_logprob[0]
if verbose:
print(f"Transcript: [{round(vad_segments[idx]['start'], 3)} --> {round(vad_segments[idx]['end'], 3)}] {text}")
segments.append(
{
"text": text,
"start": round(vad_segments[idx]['start'], 3),
"end": round(vad_segments[idx]['end'], 3)
"end": round(vad_segments[idx]['end'], 3),
"avg_logprob": avg_logprob,
}
)
@@ -285,7 +296,7 @@ class FasterWhisperPipeline(Pipeline):
def detect_language(self, audio: np.ndarray) -> str:
if audio.shape[0] < N_SAMPLES:
print("Warning: audio is shorter than 30s, language detection may be inaccurate.")
logger.warning("Audio is shorter than 30s, language detection may be inaccurate")
model_n_mels = self.model.feat_kwargs.get("feature_size")
segment = log_mel_spectrogram(audio[: N_SAMPLES],
n_mels=model_n_mels if model_n_mels is not None else 80,
@@ -294,7 +305,7 @@ class FasterWhisperPipeline(Pipeline):
results = self.model.model.detect_language(encoder_output)
language_token, language_probability = results[0][0]
language = language_token[2:-2]
print(f"Detected language: {language} ({language_probability:.2f}) in first 30s of audio...")
logger.info(f"Detected language: {language} ({language_probability:.2f}) in first 30s of audio")
return language
@@ -302,7 +313,7 @@ def load_model(
whisper_arch: str,
device: str,
device_index=0,
compute_type="float16",
compute_type="default",
asr_options: Optional[dict] = None,
language: Optional[str] = None,
vad_model: Optional[Vad]= None,
@@ -313,13 +324,16 @@ def load_model(
download_root: Optional[str] = None,
local_files_only=False,
threads=4,
use_auth_token: Optional[Union[str, bool]] = None,
) -> FasterWhisperPipeline:
"""Load a Whisper model for inference.
Args:
whisper_arch - The name of the Whisper model to load.
device - The device to load the model on.
compute_type - The compute type to use for the model.
vad_method - The vad method to use. vad_model has higher priority if is not None.
Use "default" to automatically select based on device (float16 for GPU, float32 for CPU).
vad_model - The vad model to manually assign.
vad_method - The vad method to use. vad_model has a higher priority if it is not None.
options - A dictionary of options to use for the model.
language - The language of the model. (use English for now)
model - The WhisperModel instance to use.
@@ -330,6 +344,10 @@ def load_model(
A Whisper pipeline.
"""
if compute_type == "default":
compute_type = "float16" if device == "cuda" else "float32"
logger.info(f"Compute type not specified, defaulting to {compute_type} for device {device}")
if whisper_arch.endswith(".en"):
language = "en"
@@ -339,11 +357,12 @@ def load_model(
compute_type=compute_type,
download_root=download_root,
local_files_only=local_files_only,
cpu_threads=threads)
cpu_threads=threads,
use_auth_token=use_auth_token)
if language is not None:
tokenizer = Tokenizer(model.hf_tokenizer, model.model.is_multilingual, task=task, language=language)
else:
print("No language specified, language will be first be detected for each audio file (increases inference time).")
logger.info("No language specified, language will be detected for each audio file (increases inference time)")
tokenizer = None
default_asr_options = {
@@ -401,7 +420,11 @@ def load_model(
if vad_method == "silero":
vad_model = Silero(**default_vad_options)
elif vad_method == "pyannote":
vad_model = Pyannote(torch.device(device), use_auth_token=None, **default_vad_options)
if device == 'cuda':
device_vad = f'cuda:{device_index}'
else:
device_vad = device
vad_model = Pyannote(torch.device(device_vad), token=None, **default_vad_options)
else:
raise ValueError(f"Invalid vad_method: {vad_method}")
+148 -48
View File
@@ -1,24 +1,106 @@
import numpy as np
import pandas as pd
from pyannote.audio import Pipeline
from typing import Optional, Union
from typing import Optional, Union, List, Tuple
import torch
from whisperx.audio import load_audio, SAMPLE_RATE
from whisperx.types import TranscriptionResult, AlignedTranscriptionResult
from whisperx.schema import TranscriptionResult, AlignedTranscriptionResult
from whisperx.log_utils import get_logger
logger = get_logger(__name__)
class IntervalTree:
"""
Simple interval tree for fast overlap queries using sorted array + binary search.
Uses O(n) space and provides O(log n) query time instead of O(n) linear scan.
This achieves ~228x speedup for speaker assignment in long-form content.
"""
def __init__(self, intervals: List[Tuple[float, float, str]]):
"""
Initialize the interval tree with diarization segments.
Args:
intervals: List of (start, end, speaker) tuples
"""
if not intervals:
self.starts = np.array([])
self.ends = np.array([])
self.speakers: List[str] = []
return
# Sort intervals by start time for binary search
sorted_intervals = sorted(intervals, key=lambda x: x[0])
self.starts = np.array([i[0] for i in sorted_intervals], dtype=np.float64)
self.ends = np.array([i[1] for i in sorted_intervals], dtype=np.float64)
self.speakers = [i[2] for i in sorted_intervals]
def query(self, start: float, end: float) -> List[Tuple[str, float]]:
"""
Find all intervals that overlap with [start, end] and compute intersection.
Args:
start: Query interval start time
end: Query interval end time
Returns:
List of (speaker, intersection_duration) tuples for overlapping segments
"""
if len(self.starts) == 0:
return []
# Binary search to find candidate intervals
# Only intervals with start < end could overlap
right_idx = np.searchsorted(self.starts, end, side='left')
if right_idx == 0:
return []
# Check candidates for actual overlap
candidates = slice(0, right_idx)
overlaps = (self.starts[candidates] < end) & (self.ends[candidates] > start)
results = []
for idx in np.where(overlaps)[0]:
intersection = min(self.ends[idx], end) - max(self.starts[idx], start)
if intersection > 0:
results.append((self.speakers[idx], intersection))
return results
def find_nearest(self, time: float) -> Optional[str]:
"""
Find the speaker of the nearest segment to a given time point.
Args:
time: Time point to find nearest segment for
Returns:
Speaker ID of nearest segment, or None if no segments exist
"""
if len(self.starts) == 0:
return None
# Calculate midpoints of all segments
mids = (self.starts + self.ends) / 2
nearest_idx = np.argmin(np.abs(mids - time))
return self.speakers[nearest_idx]
class DiarizationPipeline:
def __init__(
self,
model_name=None,
use_auth_token=None,
token=None,
device: Optional[Union[str, torch.device]] = "cpu",
cache_dir=None,
):
if isinstance(device, str):
device = torch.device(device)
model_config = model_name or "pyannote/speaker-diarization-3.1"
self.model = Pipeline.from_pretrained(model_config, use_auth_token=use_auth_token).to(device)
model_config = model_name or "pyannote/speaker-diarization-community-1"
logger.info(f"Loading diarization model: {model_config}")
self.model = Pipeline.from_pretrained(model_config, token=token, cache_dir=cache_dir).to(device)
def __call__(
self,
@@ -51,22 +133,15 @@ class DiarizationPipeline:
'sample_rate': SAMPLE_RATE
}
if return_embeddings:
diarization, embeddings = self.model(
audio_data,
num_speakers=num_speakers,
min_speakers=min_speakers,
max_speakers=max_speakers,
return_embeddings=True,
)
else:
diarization = self.model(
audio_data,
num_speakers=num_speakers,
min_speakers=min_speakers,
max_speakers=max_speakers,
)
embeddings = None
output = self.model(
audio_data,
num_speakers=num_speakers,
min_speakers=min_speakers,
max_speakers=max_speakers,
)
diarization = output.speaker_diarization
embeddings = output.speaker_embeddings if return_embeddings else None
diarize_df = pd.DataFrame(diarization.itertracks(yield_label=True), columns=['segment', 'label', 'speaker'])
diarize_df['start'] = diarize_df['segment'].apply(lambda x: x.start)
@@ -92,6 +167,9 @@ def assign_word_speakers(
"""
Assign speakers to words and segments in the transcript.
Uses an interval tree for O(log n) overlap queries instead of O(n) linear scan,
achieving ~228x speedup for long-form content (3+ hour podcasts).
Args:
diarize_df: Diarization dataframe from DiarizationPipeline
transcript_result: Transcription result to augment with speaker labels
@@ -101,36 +179,58 @@ def assign_word_speakers(
Returns:
Updated transcript_result with speaker assignments and optionally embeddings
"""
transcript_segments = transcript_result["segments"]
transcript_segments = transcript_result.get("segments", [])
if not transcript_segments or diarize_df is None or len(diarize_df) == 0:
return transcript_result
# Build interval tree from diarization segments for O(log n) queries
intervals = [
(row['start'], row['end'], row['speaker'])
for _, row in diarize_df.iterrows()
]
tree = IntervalTree(intervals)
for seg in transcript_segments:
# assign speaker to segment (if any)
diarize_df['intersection'] = np.minimum(diarize_df['end'], seg['end']) - np.maximum(diarize_df['start'], seg['start'])
diarize_df['union'] = np.maximum(diarize_df['end'], seg['end']) - np.minimum(diarize_df['start'], seg['start'])
# remove no hit, otherwise we look for closest (even negative intersection...)
if not fill_nearest:
dia_tmp = diarize_df[diarize_df['intersection'] > 0]
else:
dia_tmp = diarize_df
if len(dia_tmp) > 0:
# sum over speakers
speaker = dia_tmp.groupby("speaker")["intersection"].sum().sort_values(ascending=False).index[0]
seg["speaker"] = speaker
# assign speaker to words
seg_start = seg.get('start', 0.0)
seg_end = seg.get('end', 0.0)
# Query overlapping segments using interval tree
overlaps = tree.query(seg_start, seg_end)
if overlaps:
# Sum intersection durations per speaker and pick the dominant one
speaker_intersections: dict[str, float] = {}
for speaker, intersection in overlaps:
speaker_intersections[speaker] = speaker_intersections.get(speaker, 0.0) + intersection
seg['speaker'] = max(speaker_intersections.items(), key=lambda x: x[1])[0]
elif fill_nearest:
# Find nearest segment if no overlap
seg_mid = (seg_start + seg_end) / 2
nearest_speaker = tree.find_nearest(seg_mid)
if nearest_speaker:
seg['speaker'] = nearest_speaker
# Assign speaker to words
if 'words' in seg:
for word in seg['words']:
if 'start' in word:
diarize_df['intersection'] = np.minimum(diarize_df['end'], word['end']) - np.maximum(diarize_df['start'], word['start'])
diarize_df['union'] = np.maximum(diarize_df['end'], word['end']) - np.minimum(diarize_df['start'], word['start'])
# remove no hit
if not fill_nearest:
dia_tmp = diarize_df[diarize_df['intersection'] > 0]
else:
dia_tmp = diarize_df
if len(dia_tmp) > 0:
# sum over speakers
speaker = dia_tmp.groupby("speaker")["intersection"].sum().sort_values(ascending=False).index[0]
word["speaker"] = speaker
if 'start' not in word:
continue
word_start = word['start']
word_end = word.get('end', word_start)
word_overlaps = tree.query(word_start, word_end)
if word_overlaps:
speaker_intersections = {}
for speaker, intersection in word_overlaps:
speaker_intersections[speaker] = speaker_intersections.get(speaker, 0.0) + intersection
word['speaker'] = max(speaker_intersections.items(), key=lambda x: x[1])[0]
elif fill_nearest:
word_mid = (word_start + word_end) / 2
nearest_speaker = tree.find_nearest(word_mid)
if nearest_speaker:
word['speaker'] = nearest_speaker
# Add speaker embeddings to the result if provided
if speaker_embeddings is not None:
+67
View File
@@ -0,0 +1,67 @@
import logging
import sys
from typing import Optional
_LOG_FORMAT = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
_DATE_FORMAT = "%Y-%m-%d %H:%M:%S"
def setup_logging(
level: str = "info",
log_file: Optional[str] = None,
) -> None:
"""
Configure logging for WhisperX.
Args:
level: Logging level (debug, info, warning, error, critical). Default: info
log_file: Optional path to log file. If None, logs only to console.
"""
logger = logging.getLogger("whisperx")
logger.handlers.clear()
try:
log_level = getattr(logging, level.upper())
except AttributeError:
log_level = logging.WARNING
logger.setLevel(log_level)
formatter = logging.Formatter(_LOG_FORMAT, datefmt=_DATE_FORMAT)
console_handler = logging.StreamHandler(sys.stdout)
console_handler.setLevel(log_level)
console_handler.setFormatter(formatter)
logger.addHandler(console_handler)
if log_file:
try:
file_handler = logging.FileHandler(log_file)
file_handler.setLevel(log_level)
file_handler.setFormatter(formatter)
logger.addHandler(file_handler)
except (OSError) as e:
logger.warning(f"Failed to create log file '{log_file}': {e}")
logger.warning("Continuing with console logging only")
# Don't propagate to root logger to avoid duplicate messages
logger.propagate = False
def get_logger(name: str) -> logging.Logger:
"""
Get a logger instance for the given module.
Args:
name: Logger name (typically __name__ from calling module)
Returns:
Logger instance configured with WhisperX settings
"""
whisperx_logger = logging.getLogger("whisperx")
if not whisperx_logger.handlers:
setup_logging()
logger_name = "whisperx" if name == "__main__" else name
return logging.getLogger(logger_name)
+7
View File
@@ -1,5 +1,10 @@
from typing import TypedDict, Optional, List, Tuple
try:
from typing import NotRequired
except ImportError:
from typing_extensions import NotRequired
class SingleWordSegment(TypedDict):
"""
@@ -28,6 +33,7 @@ class SingleSegment(TypedDict):
start: float
end: float
text: str
avg_logprob: NotRequired[float]
class SegmentData(TypedDict):
@@ -49,6 +55,7 @@ class SingleAlignedSegment(TypedDict):
start: float
end: float
text: str
avg_logprob: NotRequired[float]
words: List[SingleWordSegment]
chars: Optional[List[SingleCharSegment]]
+16 -12
View File
@@ -10,8 +10,11 @@ from whisperx.alignment import align, load_align_model
from whisperx.asr import load_model
from whisperx.audio import load_audio
from whisperx.diarize import DiarizationPipeline, assign_word_speakers
from whisperx.types import AlignedTranscriptionResult, TranscriptionResult
from whisperx.schema import AlignedTranscriptionResult, TranscriptionResult
from whisperx.utils import LANGUAGES, TO_LANGUAGE_CODE, get_writer
from whisperx.log_utils import get_logger
logger = get_logger(__name__)
def transcribe_task(args: dict, parser: argparse.ArgumentParser):
@@ -103,6 +106,7 @@ def transcribe_task(args: dict, parser: argparse.ArgumentParser):
"no_speech_threshold": args.pop("no_speech_threshold"),
"condition_on_previous_text": False,
"initial_prompt": args.pop("initial_prompt"),
"hotwords": args.pop("hotwords"),
"suppress_tokens": [int(x) for x in args.pop("suppress_tokens").split(",")],
"suppress_numerals": args.pop("suppress_numerals"),
}
@@ -119,7 +123,6 @@ def transcribe_task(args: dict, parser: argparse.ArgumentParser):
# Part 1: VAD & ASR Loop
results = []
tmp_results = []
# model = load_model(model_name, device=device, download_root=model_dir)
model = load_model(
model_name,
@@ -138,12 +141,13 @@ def transcribe_task(args: dict, parser: argparse.ArgumentParser):
task=task,
local_files_only=model_cache_only,
threads=faster_whisper_threads,
use_auth_token=hf_token,
)
for audio_path in args.pop("audio"):
audio = load_audio(audio_path)
# >> VAD & ASR
print(">>Performing transcription...")
logger.info("Performing transcription...")
result: TranscriptionResult = model.transcribe(
audio,
batch_size=batch_size,
@@ -163,7 +167,7 @@ def transcribe_task(args: dict, parser: argparse.ArgumentParser):
tmp_results = results
results = []
align_model, align_metadata = load_align_model(
align_language, device, model_name=align_model
align_language, device, model_name=align_model, model_dir=model_dir, model_cache_only=model_cache_only
)
for result, audio_path in tmp_results:
# >> Align
@@ -176,13 +180,13 @@ def transcribe_task(args: dict, parser: argparse.ArgumentParser):
if align_model is not None and len(result["segments"]) > 0:
if result.get("language", "en") != align_metadata["language"]:
# load new language
print(
logger.info(
f"New language found ({result['language']})! Previous was ({align_metadata['language']}), loading new alignment model for new language..."
)
align_model, align_metadata = load_align_model(
result["language"], device
result["language"], device, model_dir=model_dir, model_cache_only=model_cache_only
)
print(">>Performing alignment...")
logger.info("Performing alignment...")
result: AlignedTranscriptionResult = align(
result["segments"],
align_model,
@@ -204,14 +208,14 @@ def transcribe_task(args: dict, parser: argparse.ArgumentParser):
# >> Diarize
if diarize:
if hf_token is None:
print(
"Warning, no --hf_token used, needs to be saved in environment variable, otherwise will throw error loading diarization model..."
logger.warning(
"No --hf_token provided, needs to be saved in environment variable, otherwise will throw error loading diarization model"
)
tmp_results = results
print(">>Performing diarization...")
print(">>Using model:", diarize_model_name)
logger.info("Performing diarization...")
logger.info(f"Using model: {diarize_model_name}")
results = []
diarize_model = DiarizationPipeline(model_name=diarize_model_name, use_auth_token=hf_token, device=device)
diarize_model = DiarizationPipeline(model_name=diarize_model_name, token=hf_token, device=device, cache_dir=model_dir)
for result, input_audio_path in tmp_results:
diarize_result = diarize_model(
input_audio_path,
+38 -6
View File
@@ -126,6 +126,29 @@ TO_LANGUAGE_CODE = {
LANGUAGES_WITHOUT_SPACES = ["ja", "zh"]
# Mapping of language codes to NLTK Punkt tokenizer model names
PUNKT_LANGUAGES = {
'cs': 'czech',
'da': 'danish',
'de': 'german',
'el': 'greek',
'en': 'english',
'es': 'spanish',
'et': 'estonian',
'fi': 'finnish',
'fr': 'french',
'it': 'italian',
'nl': 'dutch',
'no': 'norwegian',
'pl': 'polish',
'pt': 'portuguese',
'sl': 'slovene',
'sv': 'swedish',
'tr': 'turkish',
"ml": "malayalam",
"ru": "russian",
}
system_encoding = sys.getdefaultencoding()
if system_encoding != "utf-8":
@@ -283,10 +306,19 @@ class SubtitlesWriter(ResultWriter):
yield subtitle, times
if "words" in result["segments"][0]:
for subtitle, _ in iterate_subtitles():
sstart, ssend, speaker = _[0]
subtitle_start = self.format_timestamp(sstart)
subtitle_end = self.format_timestamp(ssend)
for subtitle, times in iterate_subtitles():
speaker = times[0][2]
# Derive cue times from word-level timestamps when available,
# falling back to segment-level times for fully unalignable subtitles.
word_starts = [w["start"] for w in subtitle if "start" in w]
word_ends = [w["end"] for w in subtitle if "end" in w]
if word_starts and word_ends:
subtitle_start = self.format_timestamp(min(word_starts))
subtitle_end = self.format_timestamp(max(word_ends))
else:
subtitle_start = self.format_timestamp(times[0][0])
subtitle_end = self.format_timestamp(times[0][1])
if result["language"] in LANGUAGES_WITHOUT_SPACES:
subtitle_text = "".join([word["word"] for word in subtitle])
else:
@@ -410,7 +442,7 @@ class WriteJSON(ResultWriter):
def get_writer(
output_format: str, output_dir: str
) -> Callable[[dict, TextIO, dict], None]:
) -> Callable[[dict, str, dict], None]:
writers = {
"txt": WriteTXT,
"vtt": WriteVTT,
@@ -425,7 +457,7 @@ def get_writer(
if output_format == "all":
all_writers = [writer(output_dir) for writer in writers.values()]
def write_all(result: dict, file: TextIO, options: dict):
def write_all(result: dict, file: str, options: dict):
for writer in all_writers:
writer(result, file, options)
+11 -8
View File
@@ -13,9 +13,12 @@ from pyannote.core import Segment
from whisperx.diarize import Segment as SegmentX
from whisperx.vads.vad import Vad
from whisperx.log_utils import get_logger
logger = get_logger(__name__)
def load_vad_model(device, vad_onset=0.500, vad_offset=0.363, use_auth_token=None, model_fp=None):
def load_vad_model(device, vad_onset=0.500, vad_offset=0.363, token=None, model_fp=None):
model_dir = torch.hub._get_torch_home()
main_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
@@ -37,7 +40,7 @@ def load_vad_model(device, vad_onset=0.500, vad_offset=0.363, use_auth_token=Non
model_bytes = open(model_fp, "rb").read()
vad_model = Model.from_pretrained(model_fp, use_auth_token=use_auth_token)
vad_model = Model.from_pretrained(model_fp, token=token)
hyperparameters = {"onset": vad_onset,
"offset": vad_offset,
"min_duration_on": 0.1,
@@ -189,11 +192,11 @@ class VoiceActivitySegmentation(VoiceActivityDetection):
self,
segmentation: PipelineModel = "pyannote/segmentation",
fscore: bool = False,
use_auth_token: Union[Text, None] = None,
token: Union[Text, None] = None,
**inference_kwargs,
):
super().__init__(segmentation=segmentation, fscore=fscore, use_auth_token=use_auth_token, **inference_kwargs)
super().__init__(segmentation=segmentation, fscore=fscore, token=token, **inference_kwargs)
def apply(self, file: AudioFile, hook: Optional[Callable] = None) -> Annotation:
"""Apply voice activity detection
@@ -231,10 +234,10 @@ class VoiceActivitySegmentation(VoiceActivityDetection):
class Pyannote(Vad):
def __init__(self, device, use_auth_token=None, model_fp=None, **kwargs):
print(">>Performing voice activity detection using Pyannote...")
def __init__(self, device, token=None, model_fp=None, **kwargs):
logger.info("Performing voice activity detection using Pyannote...")
super().__init__(kwargs['vad_onset'])
self.vad_pipeline = load_vad_model(device, use_auth_token=use_auth_token, model_fp=model_fp)
self.vad_pipeline = load_vad_model(device, token=token, model_fp=model_fp)
def __call__(self, audio: AudioFile, **kwargs):
return self.vad_pipeline(audio)
@@ -257,7 +260,7 @@ class Pyannote(Vad):
segments_list.append(SegmentX(speech_turn.start, speech_turn.end, "UNKNOWN"))
if len(segments_list) == 0:
print("No active speech found in audio")
logger.warning("No active speech found in audio")
return []
assert segments_list, "segments_list is empty."
return Vad.merge_chunks(segments_list, chunk_size, onset, offset)
+5 -2
View File
@@ -8,6 +8,9 @@ import torch
from whisperx.diarize import Segment as SegmentX
from whisperx.vads.vad import Vad
from whisperx.log_utils import get_logger
logger = get_logger(__name__)
AudioFile = Union[Text, Path, IOBase, Mapping]
@@ -15,7 +18,7 @@ AudioFile = Union[Text, Path, IOBase, Mapping]
class Silero(Vad):
# check again default values
def __init__(self, **kwargs):
print(">>Performing voice activity detection using Silero...")
logger.info("Performing voice activity detection using Silero...")
super().__init__(kwargs['vad_onset'])
self.vad_onset = kwargs['vad_onset']
@@ -60,7 +63,7 @@ class Silero(Vad):
):
assert chunk_size > 0
if len(segments_list) == 0:
print("No active speech found in audio")
logger.warning("No active speech found in audio")
return []
assert segments_list, "segments_list is empty."
return Vad.merge_chunks(segments_list, chunk_size, onset, offset)
-20
View File
@@ -52,23 +52,3 @@ class Vad:
return merged_segments
# Unused function
@staticmethod
def merge_vad(vad_arr, pad_onset=0.0, pad_offset=0.0, min_duration_off=0.0, min_duration_on=0.0):
active = Annotation()
for k, vad_t in enumerate(vad_arr):
region = Segment(vad_t[0] - pad_onset, vad_t[1] + pad_offset)
active[region, k] = 1
if pad_offset > 0.0 or pad_onset > 0.0 or min_duration_off > 0.0:
active = active.support(collar=min_duration_off)
# remove tracks shorter than min_duration_on
if min_duration_on > 0:
for segment, track in list(active.itertracks()):
if segment.duration < min_duration_on:
del active[segment, track]
active = active.for_json()
active_segs = pd.DataFrame([x['segment'] for x in active['content']])
return active_segs