Compare commits

...
41 Commits
Author SHA1 Message Date
BarabazsandGiorgio Azzinnaro 6187d25a44 feat: migrate to pyannote-audio v4 with speaker-diarization-community-1 (#1349)
* feat: upgrade pyannote-audio dependency to v4

* fix: rename use_auth_token to token for pyannote-audio v4 compatibility

* fix: add omegaconf dep

* fix: use structured output API for pyannote-audio v4 diarization

pyannote-audio 4.x no longer returns a plain Annotation (or tuple when>
return_embeddings=True). It now returns a structured output with
speaker_diarization and speaker_embeddings attributes.

* feat: switch default diarization model to speaker-diarization-community-1

Update default from pyannote/speaker-diarization-3.1 to
pyannote/speaker-diarization-community-1 (pyannote-audio v4),
add CC-BY-4.0 attribution, and update README for v4 API changes.

* fix: correct markdown link formatting for silero-vad in README.md

* chore: update version to 3.8.0


Co-authored-by: Giorgio Azzinnaro <giorgio@azzinna.ro>
2026-02-13 13:52:32 -07:00
BarabazsandClaude Opus 4.6 c4c1242edb fix: derive SRT/VTT cue times from word-level timestamps (#1347)
* fix: derive SRT/VTT cue times from word-level timestamps (#1315)

Subtitle cue start/end times were sourced from VAD segment boundaries
instead of word-level timestamps from forced alignment. This caused cues
to appear prematurely and could produce backwards chronological ordering
when VAD segments overlap.

Use min(word starts) / max(word ends) for cue timing, falling back to
segment-level times only when all words are unalignable.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: bump version to 3.7.7 in pyproject.toml

---------

Co-authored-by: Claude-Assistant <noreply@anthropic.com>
2026-02-13 12:53:49 +01:00
carlito 741ab9a2a8 Merge pull request #1343 from m-bain/fix-type-hint-decode-batch
[BugFix] Type hint fix in decode_batch List[str] not str:
2026-02-10 14:46:45 +00:00
carlito f2d853a817 Merge pull request #1342 from 1carlito/bugs
[BugFix] The variable I removed was not being used anyhwere.
2026-02-10 14:39:14 +00:00
carlos cc6f627a39 [BugFix] The variable I removed was not being used anyhwere. 2026-02-10 14:38:13 +00:00
Max Bain 863a986854 Merge pull request #1340 from RickSanchez93/main
Fix: pass no_repeat_ngram_size and repetition_penalty to CTranslate2 generate()
2026-02-10 13:02:21 +00:00
p.sacco 570b08b7be fix: add no_repeat_ngram_size and repetition_penalty options to WhisperModel 2026-02-10 10:44:58 +00:00
carlos 2b5ae8cafa [BugFix] Type hint fix in decode_batch List[str] not str: 2026-02-09 09:55:29 +00:00
Max Bain 66ada295c9 Merge pull request #1338 from Mr-Neutr0n/perf/interval-tree-speaker-assignment
Optimize assign_word_speakers with interval tree for 228x speedup
2026-02-08 18:21:32 +00:00
Mr-Neutr0n 7892a72a30 Optimize assign_word_speakers with interval tree for 228x speedup
Replace O(n*m) pandas operations with O(n log m) interval tree queries
for speaker assignment, where n = words/segments and m = diarization segments.

Performance improvement:
- 7-minute video (1185 words, 147 segments): 73.9s -> 0.32s (228x faster)
- 3-hour podcast: Minutes of processing -> Seconds

Changes:
- Add IntervalTree class using sorted array + binary search
- Refactor assign_word_speakers to use interval tree for overlap queries
- Maintain backward compatibility with same function signature
- Identical output to original implementation

The interval tree uses numpy arrays for efficient storage and binary search
(np.searchsorted) for O(log n) candidate finding, then filters candidates
for actual overlaps.

Fixes #1335
2026-02-07 12:55:26 +05:30
Barabazs 6ec4a02048 chore: drop python 3.9 support (#1328)
* chore: drop python 3.9 support

- Update requires-python to >=3.10
- Remove onnxruntime constraint (only needed for 3.9)
- Simplify numpy (remove version markers and upper bound)
- Remove pandas upper bound (<2.3.0 was for 3.9 compat)
- Remove av direct dependency (transitive via faster-whisper)

* chore(ci): remove python 3.9 from workflows

- Update build-and-release to use Python 3.10
- Remove 3.9 from python-compatibility matrix

* chore: bump version to 3.7.6
2026-01-27 10:41:40 +01:00
Barabazs 9de90e2664 fix: pin huggingface-hub<1.0.0 for pyannote-audio compatibility (#1327)
* fix: pin huggingface-hub<1.0.0 for pyannote-audio compatibility

pyannote-audio uses the deprecated `use_auth_token` parameter which was removed in huggingface-hub v1.0.0

* fix: upgrade yanked dependencies

* chore: update version to 3.7.5
2026-01-27 09:45:05 +01:00
Barabazs d32ec3e301 fix: add missing comma 2025-10-21 09:13:50 -06:00
db317c358b feat: add language-aware sentence tokenization (#1269)
* feat: add language-aware sentence tokenization

* feat: add missing punkt languages

---------

Co-authored-by: pulkit <129310466+p1kit@users.noreply.github.com>
Co-authored-by: Barabazs <31799121+Barabazs@users.noreply.github.com>
2025-10-21 15:57:26 +02:00
JulianFP 6e1d1caaf4 fix: incorrect type annotation in get_writer return value
The audio_path attribute that the __call__ method of the ResultWriter class takes is a str, not TextIO
2025-10-17 09:43:43 -06:00
Barabazs c8f7597345 feat: add hotwords argument to CLI for improved recognition of rare terms 2025-10-17 09:21:56 -06:00
Barabazs 5925e5f8c7 docs: add cuDNN troubleshooting for common issues (#1266)
* docs: add troubleshooting guide for cuDNN loading errors

* docs: add cuDNN version incompatibility troubleshooting
2025-10-16 10:56:51 +02:00
Barabazs 617835dc27 chore: upgrade torch and torchaudio dependencies to 2.8.0 2025-10-16 07:41:45 +00:00
Barabazs 92227e7412 fix: lock down torch and torchaudio versions (#1265)
* fix: update torch and torchaudio dependencies to use compatible version specifiers

* chore: update version to 3.7.3
2025-10-16 08:42:10 +02:00
PaffeandPaffe 0fa81b31f1 feat: add Swedish alignment model (#1110)
---------

Co-authored-by: Paffe <paffe@Sigge.home>
2025-10-15 08:08:18 +02:00
Barabazs 505bd9c0b5 chore: refine triton dependency to restrict installation to x86_64 Linux (#1259)
* chore: refine triton dependency to restrict installation to x86_64 Linux

* bump: update version to 3.7.2
2025-10-12 10:38:29 +02:00
Barabazs 895e5a8493 chore: update numpy dependency constraints for Python 3.13 compatibility (#1258)
* chore: update numpy dependency constraints for Python 3.13 compatibility

* bump: update version to 3.7.1
2025-10-12 10:31:44 +02:00
Barabazs a58ff9cb20 bump: update version to 3.7.0 2025-10-10 07:37:17 +00:00
Barabazs d13171cdde feat: add support for python 3.13 (#1256)
* feat: update Python version requirement to be compatible with 3.13

* feat: add Python 3.13 to compatibility matrix

* feat: update onnxruntime dependency for Python version compatibility

* fix: drop onnxruntime restriction for python >= 3.10
2025-10-10 09:36:24 +02:00
Barabazs c1c08c472f bump: update version to 3.6.0 2025-10-10 06:45:00 +00:00
Barabazs a51ae7a81a feat: add centralized logging to replace ad-hoc print statements (#1254)
* feat: add logging utility functions

* feat: add logging setup and log level argument to CLI

* feat: integrate logging across modules
2025-10-10 08:41:06 +02:00
Barabazs 3b1b9a8c4d refactor: rename types.py to schema.py to avoid stdlib conflict 2025-10-09 14:25:58 +02:00
Tomáš Hnyk 027ec57aee doc: update cpu only example (#1164) 2025-10-09 09:34:54 +02:00
3manifold 64e307cc29 chore: remove redundant variable & improve load_model function documentation (#1197)
* Remove redundant variable

* Improve function documentation
2025-10-09 09:32:02 +02:00
Adrian Wan 2663f2edb5 doc: fix diarize import in example script (#1192) 2025-10-09 09:27:07 +02:00
Barabazs c266ac5459 chore: update version to 3.5.0 2025-10-08 09:24:25 +00:00
Jim ChenandGitHub Copilot 95fecb91c8 build: upgrade PyTorch to 2.7.1 with CUDA 12.8 and multi-platform support
- feat: upgrade PyTorch to 2.7.1 and CUDA 12.8
    * Update README setup to require CUDA toolkit 12.8 instead of 12.4 (Linux and Windows)
    * Bump torch dependency from 2.6.0 to 2.7.1
    * Switch the PyTorch CUDA wheel index from cu124 to cu128


- Revert "docs: add troubleshooting section for libcudnn dependencies in README"
    * The issue of relying on two different versions of CUDNN in this project has been resolved.


- build(pyproject): relax python version and constrain package deps
    * Only download torch from PyTorch; obtain all other packages from PyPI.
    * Restrict numpy, onnxruntime, pandas to be compatible with Python 3.9


- build(pyproject): require triton 3.3.0+ for arm64 support
    * Add triton version 3.3.0 or newer to the dependencies to support arm64 architecture.

- build: skip Triton on Windows since it isn't supported
    * Add a platform marker to the triton dependency to skip it on Windows, as triton does not support Windows.

- build: configure PyTorch sources for cross-platform compatibility
    * macOS uses CPU-only PyTorch from pytorch-cpu index
    * Linux and Windows use CUDA 12.8 PyTorch from pytorch index
    * triton only installs on Linux with CUDA 12.8 support
    * Update lockfile to support multi-platform builds

- fix: restrict av to <16.0.0 for Python 3.9 compatibility
    * Add av<16.0.0 to dependencies to maintain Python 3.9 support
    * Update comment to include av in the restriction list
    * Update uv.lock accordingly

        PyAV dropped Python 3.9 support in v16.0.0:
        https://github.com/PyAV-Org/PyAV/commit/106089447cab9dc6d3ca2d1ab7e0aeff76d7a6a4


- fix: resolve PyTorch ARM64 platform compatibility issue

    * Update uv.lock to properly handle aarch64 platforms for PyTorch dependencies
    * Add resolution markers for ARM64 Linux systems to use CPU-only PyTorch builds
    * Ensure CUDA builds are only used on x86_64 platforms where supported

    Resolves ARM64 Docker build failures by preventing uv from attempting to install CUDA PyTorch on unsupported platforms

- chore: change .python-version to 3.10

---

Signed-off-by: CHEN, CHUN <jim60105@gmail.com>
Signed-off-by: Jim Chen <Jim@ChenJ.im>
Co-authored-by: GitHub Copilot <bot@ChenJ.im>
2025-10-08 11:21:28 +02:00
Nguyen Binh b1c8ac7de6 Change alignment model for Vietnamese language
Since the current model is a wav2vec2 pre-trained model for Vietnamese audio, it won't work with alignment tasks. To make it work as expected, I recommend chaining to a fine-tuned ASR version.
2025-10-03 09:41:03 +02:00
Barabazs bf150e442e feat: update Punkt tokenizer to use pre-trained model and handle missing data 2025-10-03 09:11:24 +02:00
Max Bain ed13dc8c6c recall.ai sponsor
Added information about Recall.ai's Meeting Transcription API.
2025-10-03 00:12:53 +01:00
Alex Cannan c7d31883bc Add jr, sr, and ph.d to punkt abbreviations 2025-10-01 08:59:53 +02:00
Barabazs 83afb81ac7 fix: restrict pyannote-audio version to avoid compatibility issues (#1242)
* fix: restrict pyannote-audio version to avoid compatibility issues
* chore: bump whisperX version to 3.4.3
2025-10-01 08:37:00 +02:00
2d9ce44329 fix(asr): load VAD model on correct CUDA device (#835)
fix(asr): load VAD model on correct CUDA device

Previously, the VAD sub‐model was always initialized on the default CUDA device (cuda:0), even when a higher device_index was specified. This change sets `device_vad` to `cuda:{device_index}` whenever `device == 'cuda'`, while falling back to the original `device` string for non‐CUDA cases. This ensures the VAD model is loaded on the intended GPU.


Co-authored-by: dujing <dujing@xmov.ai>
Co-authored-by: Barabazs <31799121+Barabazs@users.noreply.github.com>
2025-07-02 08:07:59 +02:00
3manifold f4261f34e9 Remove unused code in Vad class 2025-07-01 09:06:04 +02:00
Barabazs 429658d4cc chore: bump version to 3.4.2 2025-06-27 07:18:39 +00:00
Howard e0833da5dc Fix: Ensure integer tensor indexing in get_wildcard_emission() 2025-06-27 09:17:44 +02:00
19 changed files with 2662 additions and 2049 deletions

No files matched your search

+1 -1
View File
@@ -15,7 +15,7 @@ jobs:
uses: astral-sh/setup-uv@v5
with:
version: "0.5.14"
python-version: "3.9"
python-version: "3.10"
- name: Check if lockfile is up to date
run: uv lock --check
+1 -1
View File
@@ -12,7 +12,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.9", "3.10", "3.11", "3.12"]
python-version: ["3.10", "3.11", "3.12", "3.13"]
steps:
- uses: actions/checkout@v4
+1
View File
@@ -0,0 +1 @@
3.10
+76
View File
@@ -0,0 +1,76 @@
# Troubleshooting cuDNN Loading Errors
This guide helps resolve common cuDNN-related errors when running WhisperX on GPU. These issues typically occur when the system can't locate cuDNN libraries or finds conflicting versions.
## Unable to Load cuDNN Libraries
If you encounter the following error when running WhisperX:
`Unable to load any of {libcudnn_cnn.so.9.1.0, libcudnn_cnn.so.9.1, libcudnn_cnn.so.9, libcudnn_cnn.so}`
This means the cuDNN libraries are installed (via whisperx dependencies) but aren't in a location where the system's dynamic linker can find them.
### Solution 1: Add to LD_LIBRARY_PATH (Recommended)
Add this at the start of your Python script or notebook:
```python
import os
# Get current LD_LIBRARY_PATH
original = os.environ.get("LD_LIBRARY_PATH", "")
cudnn_path = "/usr/local/lib/python3.12/dist-packages/nvidia/cudnn/lib/"
os.environ['LD_LIBRARY_PATH'] = original + ":" + cudnn_path
```
**Note:** Adjust the Python version (`python3.12`) to match your environment.
### Solution 2: Symlink to LD_LIBRARY_PATH Directory
If Solution 1 didn't work and you still get the "unable to load" error, symlink the libraries to a directory that's already in your `LD_LIBRARY_PATH`:
1. Check what's in your LD_LIBRARY_PATH: `echo "$LD_LIBRARY_PATH"`
2. Assuming that there is only one path set.
Symlink the downloaded libcudnn files to that path:
`ln -s /usr/local/lib/python3.12/dist-packages/nvidia/cudnn/lib/libcudnn* "$LD_LIBRARY_PATH"/`
**Note:** If `LD_LIBRARY_PATH` contains multiple paths (separated by `:`), pick one directory and use it instead of `"$LD_LIBRARY_PATH"`. For example: `/usr/lib/x86_64-linux-gnu/`
## cuDNN Version Incompatibility
If you encounter this error:
```
RuntimeError: cuDNN version incompatibility: PyTorch was compiled against (9, 10, 2) but found runtime version (9, 2, 1)
```
This means PyTorch is finding a different cuDNN version than the one it was compiled with. **PyTorch comes bundled with its own cuDNN**, but a conflicting cuDNN in `LD_LIBRARY_PATH` is taking precedence.
### Solution: Remove Conflicting cuDNN from Path
Check if there's a conflicting cuDNN path:
```bash
echo $LD_LIBRARY_PATH
```
If you see paths pointing to older cuDNN installations (e.g., system-installed cuDNN or manually downloaded), try one of these:
**Option 1: Clear LD_LIBRARY_PATH temporarily**
```python
import os
# Let PyTorch use its bundled cuDNN
os.environ.pop('LD_LIBRARY_PATH', None)
```
**Option 2: Set LD_LIBRARY_PATH to only the correct version**
```python
import os
# Point only to the cuDNN that matches PyTorch's compiled version
os.environ['LD_LIBRARY_PATH'] = "/usr/local/lib/python3.12/dist-packages/nvidia/cudnn/lib/"
```
**Note:** This error is unlikely on a clean install. If it occurs anyway, [open an issue](https://github.com/m-bain/whisperX/issues). If you've modified system libraries or CUDA/cuDNN, the options above should help resolve most cases.
+20 -27
View File
@@ -1,5 +1,10 @@
<h1 align="center">WhisperX</h1>
## Recall.ai - Meeting Transcription API
If you’re looking for a transcription API for meetings, consider checking out [Recall.ai's Meeting Transcription API](https://www.recall.ai/product/meeting-transcription-api?utm_source=github&utm_medium=sponsorship&utm_campaign=mbain-whisperx), an API that works with Zoom, Google Meet, Microsoft Teams, and more. Recall.ai diarizes by pulling the speaker data and separate audio streams from the meeting platforms, which means 100% accurate speaker diarization with actual speaker names.
<p align="center">
<a href="https://github.com/m-bain/whisperX/stargazers">
<img src="https://img.shields.io/github/stars/m-bain/whisperX.svg?colorA=orange&colorB=orange&logo=github"
@@ -57,6 +62,15 @@ This repository provides fast automatic speech recognition (70x realtime with la
<h2 align="left" id="setup">Setup ⚙️</h2>
### 0. CUDA Installation
To use WhisperX with GPU acceleration, install the CUDA toolkit 12.8 before WhisperX. Skip this step if using only the CPU.
- For **Linux** users, install the CUDA toolkit 12.8 following this guide:
[CUDA Installation Guide for Linux](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/).
- For **Windows** users, download and install the CUDA toolkit 12.8:
[CUDA Downloads](https://developer.nvidia.com/cuda-12-8-1-download-archive).
### 1. Simple Installation (Recommended)
The easiest way to install WhisperX is through PyPi:
@@ -97,31 +111,9 @@ uv sync --all-extras --dev
You may also need to install ffmpeg, rust etc. Follow openAI instructions here https://github.com/openai/whisper#setup.
### Common Issues & Troubleshooting 🔧
#### libcudnn Dependencies (GPU Users)
If you're using WhisperX with GPU support and encounter errors like:
- `Could not load library libcudnn_ops_infer.so.8`
- `Unable to load any of {libcudnn_cnn.so.9.1.0, libcudnn_cnn.so.9.1, libcudnn_cnn.so.9, libcudnn_cnn.so}`
- `libcudnn_ops_infer.so.8: cannot open shared object file: No such file or directory`
This means your system is missing the CUDA Deep Neural Network library (cuDNN). This library is needed for GPU acceleration but isn't always installed by default.
**Install cuDNN (example for apt based systems):**
```bash
sudo apt update
sudo apt install libcudnn8 libcudnn8-dev -y
```
### Speaker Diarization
To **enable Speaker Diarization**, include your Hugging Face access token (read) that you can generate from [Here](https://huggingface.co/settings/tokens) after the `--hf_token` argument and accept the user agreement for the following models: [Segmentation](https://huggingface.co/pyannote/segmentation-3.0) and [Speaker-Diarization-3.1](https://huggingface.co/pyannote/speaker-diarization-3.1) (if you choose to use Speaker-Diarization 2.x, follow requirements [here](https://huggingface.co/pyannote/speaker-diarization) instead.)
> **Note**<br>
> As of Oct 11, 2023, there is a known issue regarding slow performance with pyannote/Speaker-Diarization-3.0 in whisperX. It is due to dependency conflicts between faster-whisper and pyannote-audio 3.0.0. Please see [this issue](https://github.com/m-bain/whisperX/issues/499) for more details and potential workarounds.
To **enable Speaker Diarization**, include your Hugging Face access token (read) that you can generate from [Here](https://huggingface.co/settings/tokens) after the `--hf_token` argument and accept the user agreement for the [speaker-diarization-community-1](https://huggingface.co/pyannote/speaker-diarization-community-1) model.
<h2 align="left" id="example">Usage 💬 (command line)</h2>
@@ -149,7 +141,7 @@ To label the transcript with speaker ID's (set number of speakers if known e.g.
To run on CPU instead of GPU (and for running on Mac OS X):
whisperx path/to/audio.wav --compute_type int8
whisperx path/to/audio.wav --compute_type int8 --device cpu
### Other languages
@@ -171,6 +163,7 @@ See more examples in other languages [here](EXAMPLES.md).
```python
import whisperx
import gc
from whisperx.diarize import DiarizationPipeline
device = "cuda"
audio_file = "audio.mp3"
@@ -201,7 +194,7 @@ print(result["segments"]) # after alignment
# import gc; import torch; gc.collect(); torch.cuda.empty_cache(); del model_a
# 3. Assign speaker labels
diarize_model = whisperx.diarize.DiarizationPipeline(use_auth_token=YOUR_HF_TOKEN, device=device)
diarize_model = DiarizationPipeline(token=YOUR_HF_TOKEN, device=device)
# add min/max number of speakers if known
diarize_segments = diarize_model(audio)
@@ -295,8 +288,8 @@ And uses the wonderful pyannote VAD / Diarization https://github.com/pyannote/py
Valuable VAD & Diarization Models from:
- [pyannote audio][https://github.com/pyannote/pyannote-audio]
- [silero vad][https://github.com/snakers4/silero-vad]
- [pyannote-audio](https://github.com/pyannote/pyannote-audio) — Speaker diarization powered by the [speaker-diarization-community-1](https://huggingface.co/pyannote/speaker-diarization-community-1) model, licensed under [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) by [pyannoteAI](https://www.pyannote.ai)
- [silero-vad](https://github.com/snakers4/silero-vad)
Great backend from [faster-whisper](https://github.com/guillaumekln/faster-whisper) and [CTranslate2](https://github.com/OpenNMT/CTranslate2)
+41 -8
View File
@@ -2,23 +2,25 @@
urls = { repository = "https://github.com/m-bain/whisperx" }
authors = [{ name = "Max Bain" }]
name = "whisperx"
version = "3.4.1"
version = "3.8.0"
description = "Time-Accurate Automatic Speech Recognition using Whisper."
readme = "README.md"
requires-python = ">=3.9, <3.13"
requires-python = ">=3.10, <3.14"
license = { text = "BSD-2-Clause" }
dependencies = [
"ctranslate2<4.5.0",
"ctranslate2>=4.5.0",
"faster-whisper>=1.1.1",
"nltk>=3.9.1",
"numpy>=2.0.2",
"onnxruntime>=1.19",
"numpy>=2.1.0",
"omegaconf>=2.3.0",
"pandas>=2.2.3",
"pyannote-audio>=3.3.2",
"torch>=2.5.1",
"torchaudio>=2.5.1",
"pyannote-audio>=4.0.0",
"huggingface-hub<1.0.0",
"torch~=2.8.0",
"torchaudio~=2.8.0",
"transformers>=4.48.0",
"triton>=3.3.0; sys_platform == 'linux' and platform_machine == 'x86_64'" # only install triton on x86_64 Linux
]
@@ -34,3 +36,34 @@ include-package-data = true
[tool.setuptools.packages.find]
where = ["."]
include = ["whisperx*"]
# torchcodec (transitive dep of pyannote-audio >=4) has no wheels for Linux aarch64
[tool.uv]
override-dependencies = [
"torchcodec>=0.6.0; (sys_platform == 'linux' and platform_machine == 'x86_64') or sys_platform == 'darwin' or sys_platform == 'win32'",
]
[tool.uv.sources]
torch = [
{ index = "pytorch-cpu", marker = "sys_platform == 'darwin'" },
{ index = "pytorch-cpu", marker = "platform_machine != 'x86_64' and sys_platform != 'darwin'" },
{ index = "pytorch", marker = "platform_machine == 'x86_64' and sys_platform != 'darwin'" },
]
torchaudio = [
{ index = "pytorch-cpu", marker = "sys_platform == 'darwin'" },
{ index = "pytorch-cpu", marker = "platform_machine != 'x86_64' and sys_platform != 'darwin'" },
{ index = "pytorch", marker = "platform_machine == 'x86_64' and sys_platform != 'darwin'" },
]
triton = [
{ index = "pytorch", marker = "sys_platform == 'linux'" },
]
[[tool.uv.index]]
name = "pytorch"
url = "https://download.pytorch.org/whl/cu128"
explicit = true
[[tool.uv.index]]
name = "pytorch-cpu"
url = "https://download.pytorch.org/whl/cpu"
explicit = true
Generated
+2148 -1878
View File
File diff suppressed because it is too large. Load diff
+26
View File
@@ -29,3 +29,29 @@ def load_audio(*args, **kwargs):
def assign_word_speakers(*args, **kwargs):
diarize = _lazy_import("diarize")
return diarize.assign_word_speakers(*args, **kwargs)
def setup_logging(*args, **kwargs):
"""
Configure logging for WhisperX.
Args:
level: Logging level (debug, info, warning, error, critical). Default: warning
log_file: Optional path to log file. If None, logs only to console.
"""
logging_module = _lazy_import("log_utils")
return logging_module.setup_logging(*args, **kwargs)
def get_logger(*args, **kwargs):
"""
Get a logger instance for the given module.
Args:
name: Logger name (typically __name__ from calling module)
Returns:
Logger instance configured with WhisperX settings
"""
logging_module = _lazy_import("log_utils")
return logging_module.get_logger(*args, **kwargs)
+14 -1
View File
@@ -6,6 +6,7 @@ import torch
from whisperx.utils import (LANGUAGES, TO_LANGUAGE_CODE, optional_float,
optional_int, str2bool)
from whisperx.log_utils import setup_logging
def cli():
@@ -23,6 +24,7 @@ def cli():
parser.add_argument("--output_dir", "-o", type=str, default=".", help="directory to save the outputs")
parser.add_argument("--output_format", "-f", type=str, default="all", choices=["all", "srt", "vtt", "txt", "tsv", "json", "aud"], help="format of the output file; if not specified, all available formats will be produced")
parser.add_argument("--verbose", type=str2bool, default=True, help="whether to print out the progress and debug messages")
parser.add_argument("--log-level", type=str, default=None, choices=["debug", "info", "warning", "error", "critical"], help="logging level (overrides --verbose if set)")
parser.add_argument("--task", type=str, default="transcribe", choices=["transcribe", "translate"], help="whether to perform X->X speech recognition ('transcribe') or X->English translation ('translate')")
parser.add_argument("--language", type=str, default=None, choices=sorted(LANGUAGES.keys()) + sorted([k.title() for k in TO_LANGUAGE_CODE.keys()]), help="language spoken in the audio, specify None to perform language detection")
@@ -43,7 +45,7 @@ def cli():
parser.add_argument("--diarize", action="store_true", help="Apply diarization to assign speaker labels to each segment/word")
parser.add_argument("--min_speakers", default=None, type=int, help="Minimum number of speakers to in audio file")
parser.add_argument("--max_speakers", default=None, type=int, help="Maximum number of speakers to in audio file")
parser.add_argument("--diarize_model", default="pyannote/speaker-diarization-3.1", type=str, help="Name of the speaker diarization model to use")
parser.add_argument("--diarize_model", default="pyannote/speaker-diarization-community-1", type=str, help="Name of the speaker diarization model to use")
parser.add_argument("--speaker_embeddings", action="store_true", help="Include speaker embeddings in JSON output (only works with --diarize)")
parser.add_argument("--temperature", type=float, default=0, help="temperature to use for sampling")
@@ -56,6 +58,7 @@ def cli():
parser.add_argument("--suppress_numerals", action="store_true", help="whether to suppress numeric symbols and currency symbols during sampling, since wav2vec2 cannot align them correctly")
parser.add_argument("--initial_prompt", type=str, default=None, help="optional text to provide as a prompt for the first window.")
parser.add_argument("--hotwords", type=str, default=None, help="hotwords/hint phrases to the model (e.g. \"WhisperX, PyAnnote, GPU\"); improves recognition of rare/technical terms")
parser.add_argument("--condition_on_previous_text", type=str2bool, default=False, help="if True, provide the previous output of the model as a prompt for the next window; disabling may make the text inconsistent across windows, but the model becomes less prone to getting stuck in a failure loop")
parser.add_argument("--fp16", type=str2bool, default=True, help="whether to perform inference in fp16; True by default")
@@ -80,6 +83,16 @@ def cli():
args = parser.parse_args().__dict__
log_level = args.get("log_level")
verbose = args.get("verbose")
if log_level is not None:
setup_logging(level=log_level)
elif verbose:
setup_logging(level="info")
else:
setup_logging(level="warning")
from whisperx.transcribe import transcribe_task
transcribe_task(args, parser)
+34 -26
View File
@@ -14,17 +14,19 @@ import torchaudio
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
from whisperx.audio import SAMPLE_RATE, load_audio
from whisperx.utils import interpolate_nans
from whisperx.types import (
from whisperx.utils import interpolate_nans, PUNKT_LANGUAGES
from whisperx.schema import (
AlignedTranscriptionResult,
SingleSegment,
SingleAlignedSegment,
SingleWordSegment,
SegmentData,
)
from nltk.tokenize.punkt import PunktSentenceTokenizer, PunktParameters
import nltk
from nltk.data import load as nltk_load
from whisperx.log_utils import get_logger
PUNKT_ABBREVIATIONS = ['dr', 'vs', 'mr', 'mrs', 'prof']
logger = get_logger(__name__)
LANGUAGES_WITHOUT_SPACES = ["ja", "zh"]
@@ -53,7 +55,7 @@ DEFAULT_ALIGN_MODELS_HF = {
"tr": "mpoyraz/wav2vec2-xls-r-300m-cv7-turkish",
"da": "saattrupdan/wav2vec2-xls-r-300m-ftspeech",
"he": "imvladikon/wav2vec2-xls-r-300m-hebrew",
"vi": 'nguyenvulebinh/wav2vec2-base-vi',
"vi": 'nguyenvulebinh/wav2vec2-base-vi-vlsp2020',
"ko": "kresnik/wav2vec2-large-xlsr-korean",
"ur": "kingabzpro/wav2vec2-large-xls-r-300m-Urdu",
"te": "anuragshas/wav2vec2-large-xlsr-53-telugu",
@@ -71,6 +73,7 @@ DEFAULT_ALIGN_MODELS_HF = {
"ka": "xsway/wav2vec2-large-xlsr-georgian",
"lv": "jimregan/wav2vec2-large-xlsr-latvian-cv",
"tl": "Khalsuu/filipino-wav2vec2-l-xls-r-300m-official",
"sv": "KBLab/wav2vec2-large-voxrex-swedish",
}
@@ -82,8 +85,9 @@ def load_align_model(language_code: str, device: str, model_name: Optional[str]
elif language_code in DEFAULT_ALIGN_MODELS_HF:
model_name = DEFAULT_ALIGN_MODELS_HF[language_code]
else:
print(f"There is no default alignment model set for this language ({language_code}).\
Please find a wav2vec2.0 model finetuned on this language in https://huggingface.co/models, then pass the model name in --align_model [MODEL_NAME]")
logger.error(f"No default alignment model for language: {language_code}. "
f"Please find a wav2vec2.0 model finetuned on this language at https://huggingface.co/models, "
f"then pass the model name via --align_model [MODEL_NAME]")
raise ValueError(f"No default align-model for language: {language_code}")
if model_name in torchaudio.pipelines.__all__:
@@ -124,14 +128,14 @@ def align(
"""
Align phoneme recognition predictions to known transcription.
"""
if not torch.is_tensor(audio):
if isinstance(audio, str):
audio = load_audio(audio)
audio = torch.from_numpy(audio)
if len(audio.shape) == 1:
audio = audio.unsqueeze(0)
MAX_DURATION = audio.shape[1] / SAMPLE_RATE
model_dictionary = align_model_metadata["dictionary"]
@@ -148,7 +152,7 @@ def align(
base_progress = ((sdx + 1) / total_segments) * 100
percent_complete = (50 + base_progress / 2) if combined_progress else base_progress
print(f"Progress: {percent_complete:.2f}%...")
num_leading = len(segment["text"]) - len(segment["text"].lstrip())
num_trailing = len(segment["text"]) - len(segment["text"].rstrip())
text = segment["text"]
@@ -165,7 +169,7 @@ def align(
# wav2vec2 models use "|" character to represent spaces
if model_lang not in LANGUAGES_WITHOUT_SPACES:
char_ = char_.replace(" ", "|")
# ignore whitespace at beginning and end of transcript
if cdx < num_leading:
pass
@@ -187,10 +191,14 @@ def align(
# index for placeholder
clean_wdx.append(wdx)
punkt_param = PunktParameters()
punkt_param.abbrev_types = set(PUNKT_ABBREVIATIONS)
sentence_splitter = PunktSentenceTokenizer(punkt_param)
# Use language-specific Punkt model if available otherwise we fallback to English.
punkt_lang = PUNKT_LANGUAGES.get(model_lang, 'english')
try:
sentence_splitter = nltk_load(f'tokenizers/punkt_tab/{punkt_lang}.pickle')
except LookupError:
nltk.download('punkt_tab', quiet=True)
sentence_splitter = nltk_load(f'tokenizers/punkt_tab/{punkt_lang}.pickle')
sentence_spans = list(sentence_splitter.span_tokenize(text))
segment_data[sdx] = {
@@ -199,12 +207,12 @@ def align(
"clean_wdx": clean_wdx,
"sentence_spans": sentence_spans
}
aligned_segments: List[SingleAlignedSegment] = []
# 2. Get prediction matrix from alignment model & align
for sdx, segment in enumerate(transcript):
t1 = segment["start"]
t2 = segment["end"]
text = segment["text"]
@@ -222,12 +230,12 @@ def align(
# check we can align
if len(segment_data[sdx]["clean_char"]) == 0:
print(f'Failed to align segment ("{segment["text"]}"): no characters in this segment found in model dictionary, resorting to original...')
logger.warning(f'Failed to align segment ("{segment["text"]}"): no characters in this segment found in model dictionary, resorting to original')
aligned_segments.append(aligned_seg)
continue
if t1 >= MAX_DURATION:
print(f'Failed to align segment ("{segment["text"]}"): original start time longer than audio duration, skipping...')
logger.warning(f'Failed to align segment ("{segment["text"]}"): original start time longer than audio duration, skipping')
aligned_segments.append(aligned_seg)
continue
@@ -247,7 +255,7 @@ def align(
)
else:
lengths = None
with torch.inference_mode():
if model_type == "torchaudio":
emissions, _ = model(waveform_segment.to(device), lengths=lengths)
@@ -269,7 +277,7 @@ def align(
path = backtrack_beam(trellis, emission, tokens, blank_id, beam_width=2)
if path is None:
print(f'Failed to align segment ("{segment["text"]}"): backtrack failed, resorting to original...')
logger.warning(f'Failed to align segment ("{segment["text"]}"): backtrack failed, resorting to original')
aligned_segments.append(aligned_seg)
continue
@@ -304,7 +312,7 @@ def align(
word_idx += 1
elif cdx == len(text) - 1 or text[cdx+1] == " ":
word_idx += 1
char_segments_arr = pd.DataFrame(char_segments_arr)
aligned_subsegments = []
@@ -333,7 +341,7 @@ def align(
word_end = word_chars["end"].max()
word_score = round(word_chars["score"].mean(), 3)
# -1 indicates unalignable
# -1 indicates unalignable
word_segment = {"word": word_text}
if not np.isnan(word_start):
@@ -344,7 +352,7 @@ def align(
word_segment["score"] = word_score
sentence_words.append(word_segment)
aligned_subsegments.append({
"text": sentence_text,
"start": sentence_start,
@@ -424,7 +432,7 @@ def get_wildcard_emission(frame_emission, tokens, blank_id):
wildcard_mask = (tokens == -1)
# Get scores for non-wildcard positions
regular_scores = frame_emission[tokens.clamp(min=0)] # clamp to avoid -1 index
regular_scores = frame_emission[tokens.clamp(min=0).long()] # clamp to avoid -1 index
# Create a mask and compute the maximum value without modifying frame_emission
max_valid_score = frame_emission.clone() # Create a copy
+19 -13
View File
@@ -12,8 +12,11 @@ from transformers import Pipeline
from transformers.pipelines.pt_utils import PipelineIterator
from whisperx.audio import N_SAMPLES, SAMPLE_RATE, load_audio, log_mel_spectrogram
from whisperx.types import SingleSegment, TranscriptionResult
from whisperx.schema import SingleSegment, TranscriptionResult
from whisperx.vads import Vad, Silero, Pyannote
from whisperx.log_utils import get_logger
logger = get_logger(__name__)
def find_numeral_symbol_tokens(tokenizer):
@@ -55,11 +58,7 @@ class WhisperModel(faster_whisper.WhisperModel):
)
encoder_output = self.encode(features)
max_initial_timestamp_index = int(
round(options.max_initial_timestamp / self.time_precision)
)
result = self.model.generate(
encoder_output,
[prompt] * batch_size,
@@ -69,11 +68,13 @@ class WhisperModel(faster_whisper.WhisperModel):
max_length=self.max_length,
suppress_blank=options.suppress_blank,
suppress_tokens=options.suppress_tokens,
no_repeat_ngram_size=options.no_repeat_ngram_size,
repetition_penalty=options.repetition_penalty,
)
tokens_batch = [x.sequences_ids[0] for x in result]
def decode_batch(tokens: List[List[int]]) -> str:
def decode_batch(tokens: List[List[int]]) -> List[str]:
res = []
for tk in tokens:
res.append([token for token in tk if token < tokenizer.eot])
@@ -247,7 +248,7 @@ class FasterWhisperPipeline(Pipeline):
if self.suppress_numerals:
previous_suppress_tokens = self.options.suppress_tokens
numeral_symbol_tokens = find_numeral_symbol_tokens(self.tokenizer)
print(f"Suppressing numeral and symbol tokens")
logger.info("Suppressing numeral and symbol tokens")
new_suppressed_tokens = numeral_symbol_tokens + self.options.suppress_tokens
new_suppressed_tokens = list(set(new_suppressed_tokens))
self.options = replace(self.options, suppress_tokens=new_suppressed_tokens)
@@ -285,7 +286,7 @@ class FasterWhisperPipeline(Pipeline):
def detect_language(self, audio: np.ndarray) -> str:
if audio.shape[0] < N_SAMPLES:
print("Warning: audio is shorter than 30s, language detection may be inaccurate.")
logger.warning("Audio is shorter than 30s, language detection may be inaccurate")
model_n_mels = self.model.feat_kwargs.get("feature_size")
segment = log_mel_spectrogram(audio[: N_SAMPLES],
n_mels=model_n_mels if model_n_mels is not None else 80,
@@ -294,7 +295,7 @@ class FasterWhisperPipeline(Pipeline):
results = self.model.model.detect_language(encoder_output)
language_token, language_probability = results[0][0]
language = language_token[2:-2]
print(f"Detected language: {language} ({language_probability:.2f}) in first 30s of audio...")
logger.info(f"Detected language: {language} ({language_probability:.2f}) in first 30s of audio")
return language
@@ -319,7 +320,8 @@ def load_model(
whisper_arch - The name of the Whisper model to load.
device - The device to load the model on.
compute_type - The compute type to use for the model.
vad_method - The vad method to use. vad_model has higher priority if is not None.
vad_model - The vad model to manually assign.
vad_method - The vad method to use. vad_model has a higher priority if it is not None.
options - A dictionary of options to use for the model.
language - The language of the model. (use English for now)
model - The WhisperModel instance to use.
@@ -343,7 +345,7 @@ def load_model(
if language is not None:
tokenizer = Tokenizer(model.hf_tokenizer, model.model.is_multilingual, task=task, language=language)
else:
print("No language specified, language will be first be detected for each audio file (increases inference time).")
logger.info("No language specified, language will be detected for each audio file (increases inference time)")
tokenizer = None
default_asr_options = {
@@ -401,7 +403,11 @@ def load_model(
if vad_method == "silero":
vad_model = Silero(**default_vad_options)
elif vad_method == "pyannote":
vad_model = Pyannote(torch.device(device), use_auth_token=None, **default_vad_options)
if device == 'cuda':
device_vad = f'cuda:{device_index}'
else:
device_vad = device
vad_model = Pyannote(torch.device(device_vad), token=None, **default_vad_options)
else:
raise ValueError(f"Invalid vad_method: {vad_method}")
+147 -48
View File
@@ -1,24 +1,105 @@
import numpy as np
import pandas as pd
from pyannote.audio import Pipeline
from typing import Optional, Union
from typing import Optional, Union, List, Tuple
import torch
from whisperx.audio import load_audio, SAMPLE_RATE
from whisperx.types import TranscriptionResult, AlignedTranscriptionResult
from whisperx.schema import TranscriptionResult, AlignedTranscriptionResult
from whisperx.log_utils import get_logger
logger = get_logger(__name__)
class IntervalTree:
"""
Simple interval tree for fast overlap queries using sorted array + binary search.
Uses O(n) space and provides O(log n) query time instead of O(n) linear scan.
This achieves ~228x speedup for speaker assignment in long-form content.
"""
def __init__(self, intervals: List[Tuple[float, float, str]]):
"""
Initialize the interval tree with diarization segments.
Args:
intervals: List of (start, end, speaker) tuples
"""
if not intervals:
self.starts = np.array([])
self.ends = np.array([])
self.speakers: List[str] = []
return
# Sort intervals by start time for binary search
sorted_intervals = sorted(intervals, key=lambda x: x[0])
self.starts = np.array([i[0] for i in sorted_intervals], dtype=np.float64)
self.ends = np.array([i[1] for i in sorted_intervals], dtype=np.float64)
self.speakers = [i[2] for i in sorted_intervals]
def query(self, start: float, end: float) -> List[Tuple[str, float]]:
"""
Find all intervals that overlap with [start, end] and compute intersection.
Args:
start: Query interval start time
end: Query interval end time
Returns:
List of (speaker, intersection_duration) tuples for overlapping segments
"""
if len(self.starts) == 0:
return []
# Binary search to find candidate intervals
# Only intervals with start < end could overlap
right_idx = np.searchsorted(self.starts, end, side='left')
if right_idx == 0:
return []
# Check candidates for actual overlap
candidates = slice(0, right_idx)
overlaps = (self.starts[candidates] < end) & (self.ends[candidates] > start)
results = []
for idx in np.where(overlaps)[0]:
intersection = min(self.ends[idx], end) - max(self.starts[idx], start)
if intersection > 0:
results.append((self.speakers[idx], intersection))
return results
def find_nearest(self, time: float) -> Optional[str]:
"""
Find the speaker of the nearest segment to a given time point.
Args:
time: Time point to find nearest segment for
Returns:
Speaker ID of nearest segment, or None if no segments exist
"""
if len(self.starts) == 0:
return None
# Calculate midpoints of all segments
mids = (self.starts + self.ends) / 2
nearest_idx = np.argmin(np.abs(mids - time))
return self.speakers[nearest_idx]
class DiarizationPipeline:
def __init__(
self,
model_name=None,
use_auth_token=None,
token=None,
device: Optional[Union[str, torch.device]] = "cpu",
):
if isinstance(device, str):
device = torch.device(device)
model_config = model_name or "pyannote/speaker-diarization-3.1"
self.model = Pipeline.from_pretrained(model_config, use_auth_token=use_auth_token).to(device)
model_config = model_name or "pyannote/speaker-diarization-community-1"
logger.info(f"Loading diarization model: {model_config}")
self.model = Pipeline.from_pretrained(model_config, token=token).to(device)
def __call__(
self,
@@ -51,22 +132,15 @@ class DiarizationPipeline:
'sample_rate': SAMPLE_RATE
}
if return_embeddings:
diarization, embeddings = self.model(
audio_data,
num_speakers=num_speakers,
min_speakers=min_speakers,
max_speakers=max_speakers,
return_embeddings=True,
)
else:
diarization = self.model(
audio_data,
num_speakers=num_speakers,
min_speakers=min_speakers,
max_speakers=max_speakers,
)
embeddings = None
output = self.model(
audio_data,
num_speakers=num_speakers,
min_speakers=min_speakers,
max_speakers=max_speakers,
)
diarization = output.speaker_diarization
embeddings = output.speaker_embeddings if return_embeddings else None
diarize_df = pd.DataFrame(diarization.itertracks(yield_label=True), columns=['segment', 'label', 'speaker'])
diarize_df['start'] = diarize_df['segment'].apply(lambda x: x.start)
@@ -92,6 +166,9 @@ def assign_word_speakers(
"""
Assign speakers to words and segments in the transcript.
Uses an interval tree for O(log n) overlap queries instead of O(n) linear scan,
achieving ~228x speedup for long-form content (3+ hour podcasts).
Args:
diarize_df: Diarization dataframe from DiarizationPipeline
transcript_result: Transcription result to augment with speaker labels
@@ -101,36 +178,58 @@ def assign_word_speakers(
Returns:
Updated transcript_result with speaker assignments and optionally embeddings
"""
transcript_segments = transcript_result["segments"]
transcript_segments = transcript_result.get("segments", [])
if not transcript_segments or diarize_df is None or len(diarize_df) == 0:
return transcript_result
# Build interval tree from diarization segments for O(log n) queries
intervals = [
(row['start'], row['end'], row['speaker'])
for _, row in diarize_df.iterrows()
]
tree = IntervalTree(intervals)
for seg in transcript_segments:
# assign speaker to segment (if any)
diarize_df['intersection'] = np.minimum(diarize_df['end'], seg['end']) - np.maximum(diarize_df['start'], seg['start'])
diarize_df['union'] = np.maximum(diarize_df['end'], seg['end']) - np.minimum(diarize_df['start'], seg['start'])
# remove no hit, otherwise we look for closest (even negative intersection...)
if not fill_nearest:
dia_tmp = diarize_df[diarize_df['intersection'] > 0]
else:
dia_tmp = diarize_df
if len(dia_tmp) > 0:
# sum over speakers
speaker = dia_tmp.groupby("speaker")["intersection"].sum().sort_values(ascending=False).index[0]
seg["speaker"] = speaker
# assign speaker to words
seg_start = seg.get('start', 0.0)
seg_end = seg.get('end', 0.0)
# Query overlapping segments using interval tree
overlaps = tree.query(seg_start, seg_end)
if overlaps:
# Sum intersection durations per speaker and pick the dominant one
speaker_intersections: dict[str, float] = {}
for speaker, intersection in overlaps:
speaker_intersections[speaker] = speaker_intersections.get(speaker, 0.0) + intersection
seg['speaker'] = max(speaker_intersections.items(), key=lambda x: x[1])[0]
elif fill_nearest:
# Find nearest segment if no overlap
seg_mid = (seg_start + seg_end) / 2
nearest_speaker = tree.find_nearest(seg_mid)
if nearest_speaker:
seg['speaker'] = nearest_speaker
# Assign speaker to words
if 'words' in seg:
for word in seg['words']:
if 'start' in word:
diarize_df['intersection'] = np.minimum(diarize_df['end'], word['end']) - np.maximum(diarize_df['start'], word['start'])
diarize_df['union'] = np.maximum(diarize_df['end'], word['end']) - np.minimum(diarize_df['start'], word['start'])
# remove no hit
if not fill_nearest:
dia_tmp = diarize_df[diarize_df['intersection'] > 0]
else:
dia_tmp = diarize_df
if len(dia_tmp) > 0:
# sum over speakers
speaker = dia_tmp.groupby("speaker")["intersection"].sum().sort_values(ascending=False).index[0]
word["speaker"] = speaker
if 'start' not in word:
continue
word_start = word['start']
word_end = word.get('end', word_start)
word_overlaps = tree.query(word_start, word_end)
if word_overlaps:
speaker_intersections = {}
for speaker, intersection in word_overlaps:
speaker_intersections[speaker] = speaker_intersections.get(speaker, 0.0) + intersection
word['speaker'] = max(speaker_intersections.items(), key=lambda x: x[1])[0]
elif fill_nearest:
word_mid = (word_start + word_end) / 2
nearest_speaker = tree.find_nearest(word_mid)
if nearest_speaker:
word['speaker'] = nearest_speaker
# Add speaker embeddings to the result if provided
if speaker_embeddings is not None:
+67
View File
@@ -0,0 +1,67 @@
import logging
import sys
from typing import Optional
_LOG_FORMAT = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
_DATE_FORMAT = "%Y-%m-%d %H:%M:%S"
def setup_logging(
level: str = "info",
log_file: Optional[str] = None,
) -> None:
"""
Configure logging for WhisperX.
Args:
level: Logging level (debug, info, warning, error, critical). Default: info
log_file: Optional path to log file. If None, logs only to console.
"""
logger = logging.getLogger("whisperx")
logger.handlers.clear()
try:
log_level = getattr(logging, level.upper())
except AttributeError:
log_level = logging.WARNING
logger.setLevel(log_level)
formatter = logging.Formatter(_LOG_FORMAT, datefmt=_DATE_FORMAT)
console_handler = logging.StreamHandler(sys.stdout)
console_handler.setLevel(log_level)
console_handler.setFormatter(formatter)
logger.addHandler(console_handler)
if log_file:
try:
file_handler = logging.FileHandler(log_file)
file_handler.setLevel(log_level)
file_handler.setFormatter(formatter)
logger.addHandler(file_handler)
except (OSError) as e:
logger.warning(f"Failed to create log file '{log_file}': {e}")
logger.warning("Continuing with console logging only")
# Don't propagate to root logger to avoid duplicate messages
logger.propagate = False
def get_logger(name: str) -> logging.Logger:
"""
Get a logger instance for the given module.
Args:
name: Logger name (typically __name__ from calling module)
Returns:
Logger instance configured with WhisperX settings
"""
whisperx_logger = logging.getLogger("whisperx")
if not whisperx_logger.handlers:
setup_logging()
logger_name = "whisperx" if name == "__main__" else name
return logging.getLogger(logger_name)
File renamed without changes.
+13 -10
View File
@@ -10,8 +10,11 @@ from whisperx.alignment import align, load_align_model
from whisperx.asr import load_model
from whisperx.audio import load_audio
from whisperx.diarize import DiarizationPipeline, assign_word_speakers
from whisperx.types import AlignedTranscriptionResult, TranscriptionResult
from whisperx.schema import AlignedTranscriptionResult, TranscriptionResult
from whisperx.utils import LANGUAGES, TO_LANGUAGE_CODE, get_writer
from whisperx.log_utils import get_logger
logger = get_logger(__name__)
def transcribe_task(args: dict, parser: argparse.ArgumentParser):
@@ -103,6 +106,7 @@ def transcribe_task(args: dict, parser: argparse.ArgumentParser):
"no_speech_threshold": args.pop("no_speech_threshold"),
"condition_on_previous_text": False,
"initial_prompt": args.pop("initial_prompt"),
"hotwords": args.pop("hotwords"),
"suppress_tokens": [int(x) for x in args.pop("suppress_tokens").split(",")],
"suppress_numerals": args.pop("suppress_numerals"),
}
@@ -119,7 +123,6 @@ def transcribe_task(args: dict, parser: argparse.ArgumentParser):
# Part 1: VAD & ASR Loop
results = []
tmp_results = []
# model = load_model(model_name, device=device, download_root=model_dir)
model = load_model(
model_name,
@@ -143,7 +146,7 @@ def transcribe_task(args: dict, parser: argparse.ArgumentParser):
for audio_path in args.pop("audio"):
audio = load_audio(audio_path)
# >> VAD & ASR
print(">>Performing transcription...")
logger.info("Performing transcription...")
result: TranscriptionResult = model.transcribe(
audio,
batch_size=batch_size,
@@ -176,13 +179,13 @@ def transcribe_task(args: dict, parser: argparse.ArgumentParser):
if align_model is not None and len(result["segments"]) > 0:
if result.get("language", "en") != align_metadata["language"]:
# load new language
print(
logger.info(
f"New language found ({result['language']})! Previous was ({align_metadata['language']}), loading new alignment model for new language..."
)
align_model, align_metadata = load_align_model(
result["language"], device
)
print(">>Performing alignment...")
logger.info("Performing alignment...")
result: AlignedTranscriptionResult = align(
result["segments"],
align_model,
@@ -204,14 +207,14 @@ def transcribe_task(args: dict, parser: argparse.ArgumentParser):
# >> Diarize
if diarize:
if hf_token is None:
print(
"Warning, no --hf_token used, needs to be saved in environment variable, otherwise will throw error loading diarization model..."
logger.warning(
"No --hf_token provided, needs to be saved in environment variable, otherwise will throw error loading diarization model"
)
tmp_results = results
print(">>Performing diarization...")
print(">>Using model:", diarize_model_name)
logger.info("Performing diarization...")
logger.info(f"Using model: {diarize_model_name}")
results = []
diarize_model = DiarizationPipeline(model_name=diarize_model_name, use_auth_token=hf_token, device=device)
diarize_model = DiarizationPipeline(model_name=diarize_model_name, token=hf_token, device=device)
for result, input_audio_path in tmp_results:
diarize_result = diarize_model(
input_audio_path,
+38 -6
View File
@@ -126,6 +126,29 @@ TO_LANGUAGE_CODE = {
LANGUAGES_WITHOUT_SPACES = ["ja", "zh"]
# Mapping of language codes to NLTK Punkt tokenizer model names
PUNKT_LANGUAGES = {
'cs': 'czech',
'da': 'danish',
'de': 'german',
'el': 'greek',
'en': 'english',
'es': 'spanish',
'et': 'estonian',
'fi': 'finnish',
'fr': 'french',
'it': 'italian',
'nl': 'dutch',
'no': 'norwegian',
'pl': 'polish',
'pt': 'portuguese',
'sl': 'slovene',
'sv': 'swedish',
'tr': 'turkish',
"ml": "malayalam",
"ru": "russian",
}
system_encoding = sys.getdefaultencoding()
if system_encoding != "utf-8":
@@ -283,10 +306,19 @@ class SubtitlesWriter(ResultWriter):
yield subtitle, times
if "words" in result["segments"][0]:
for subtitle, _ in iterate_subtitles():
sstart, ssend, speaker = _[0]
subtitle_start = self.format_timestamp(sstart)
subtitle_end = self.format_timestamp(ssend)
for subtitle, times in iterate_subtitles():
speaker = times[0][2]
# Derive cue times from word-level timestamps when available,
# falling back to segment-level times for fully unalignable subtitles.
word_starts = [w["start"] for w in subtitle if "start" in w]
word_ends = [w["end"] for w in subtitle if "end" in w]
if word_starts and word_ends:
subtitle_start = self.format_timestamp(min(word_starts))
subtitle_end = self.format_timestamp(max(word_ends))
else:
subtitle_start = self.format_timestamp(times[0][0])
subtitle_end = self.format_timestamp(times[0][1])
if result["language"] in LANGUAGES_WITHOUT_SPACES:
subtitle_text = "".join([word["word"] for word in subtitle])
else:
@@ -410,7 +442,7 @@ class WriteJSON(ResultWriter):
def get_writer(
output_format: str, output_dir: str
) -> Callable[[dict, TextIO, dict], None]:
) -> Callable[[dict, str, dict], None]:
writers = {
"txt": WriteTXT,
"vtt": WriteVTT,
@@ -425,7 +457,7 @@ def get_writer(
if output_format == "all":
all_writers = [writer(output_dir) for writer in writers.values()]
def write_all(result: dict, file: TextIO, options: dict):
def write_all(result: dict, file: str, options: dict):
for writer in all_writers:
writer(result, file, options)
+11 -8
View File
@@ -13,9 +13,12 @@ from pyannote.core import Segment
from whisperx.diarize import Segment as SegmentX
from whisperx.vads.vad import Vad
from whisperx.log_utils import get_logger
logger = get_logger(__name__)
def load_vad_model(device, vad_onset=0.500, vad_offset=0.363, use_auth_token=None, model_fp=None):
def load_vad_model(device, vad_onset=0.500, vad_offset=0.363, token=None, model_fp=None):
model_dir = torch.hub._get_torch_home()
main_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
@@ -37,7 +40,7 @@ def load_vad_model(device, vad_onset=0.500, vad_offset=0.363, use_auth_token=Non
model_bytes = open(model_fp, "rb").read()
vad_model = Model.from_pretrained(model_fp, use_auth_token=use_auth_token)
vad_model = Model.from_pretrained(model_fp, token=token)
hyperparameters = {"onset": vad_onset,
"offset": vad_offset,
"min_duration_on": 0.1,
@@ -189,11 +192,11 @@ class VoiceActivitySegmentation(VoiceActivityDetection):
self,
segmentation: PipelineModel = "pyannote/segmentation",
fscore: bool = False,
use_auth_token: Union[Text, None] = None,
token: Union[Text, None] = None,
**inference_kwargs,
):
super().__init__(segmentation=segmentation, fscore=fscore, use_auth_token=use_auth_token, **inference_kwargs)
super().__init__(segmentation=segmentation, fscore=fscore, token=token, **inference_kwargs)
def apply(self, file: AudioFile, hook: Optional[Callable] = None) -> Annotation:
"""Apply voice activity detection
@@ -231,10 +234,10 @@ class VoiceActivitySegmentation(VoiceActivityDetection):
class Pyannote(Vad):
def __init__(self, device, use_auth_token=None, model_fp=None, **kwargs):
print(">>Performing voice activity detection using Pyannote...")
def __init__(self, device, token=None, model_fp=None, **kwargs):
logger.info("Performing voice activity detection using Pyannote...")
super().__init__(kwargs['vad_onset'])
self.vad_pipeline = load_vad_model(device, use_auth_token=use_auth_token, model_fp=model_fp)
self.vad_pipeline = load_vad_model(device, token=token, model_fp=model_fp)
def __call__(self, audio: AudioFile, **kwargs):
return self.vad_pipeline(audio)
@@ -257,7 +260,7 @@ class Pyannote(Vad):
segments_list.append(SegmentX(speech_turn.start, speech_turn.end, "UNKNOWN"))
if len(segments_list) == 0:
print("No active speech found in audio")
logger.warning("No active speech found in audio")
return []
assert segments_list, "segments_list is empty."
return Vad.merge_chunks(segments_list, chunk_size, onset, offset)
+5 -2
View File
@@ -8,6 +8,9 @@ import torch
from whisperx.diarize import Segment as SegmentX
from whisperx.vads.vad import Vad
from whisperx.log_utils import get_logger
logger = get_logger(__name__)
AudioFile = Union[Text, Path, IOBase, Mapping]
@@ -15,7 +18,7 @@ AudioFile = Union[Text, Path, IOBase, Mapping]
class Silero(Vad):
# check again default values
def __init__(self, **kwargs):
print(">>Performing voice activity detection using Silero...")
logger.info("Performing voice activity detection using Silero...")
super().__init__(kwargs['vad_onset'])
self.vad_onset = kwargs['vad_onset']
@@ -60,7 +63,7 @@ class Silero(Vad):
):
assert chunk_size > 0
if len(segments_list) == 0:
print("No active speech found in audio")
logger.warning("No active speech found in audio")
return []
assert segments_list, "segments_list is empty."
return Vad.merge_chunks(segments_list, chunk_size, onset, offset)
-20
View File
@@ -52,23 +52,3 @@ class Vad:
return merged_segments
# Unused function
@staticmethod
def merge_vad(vad_arr, pad_onset=0.0, pad_offset=0.0, min_duration_off=0.0, min_duration_on=0.0):
active = Annotation()
for k, vad_t in enumerate(vad_arr):
region = Segment(vad_t[0] - pad_onset, vad_t[1] + pad_offset)
active[region, k] = 1
if pad_offset > 0.0 or pad_onset > 0.0 or min_duration_off > 0.0:
active = active.support(collar=min_duration_off)
# remove tracks shorter than min_duration_on
if min_duration_on > 0:
for segment, track in list(active.itertracks()):
if segment.duration < min_duration_on:
del active[segment, track]
active = active.for_json()
active_segs = pd.DataFrame([x['segment'] for x in active['content']])
return active_segs