Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6ec4a02048 | ||
|
|
9de90e2664 | ||
|
|
d32ec3e301 | ||
|
|
db317c358b | ||
|
|
6e1d1caaf4 | ||
|
|
c8f7597345 | ||
|
|
5925e5f8c7 | ||
|
|
617835dc27 | ||
|
|
92227e7412 | ||
|
|
0fa81b31f1 | ||
|
|
505bd9c0b5 | ||
|
|
895e5a8493 | ||
|
|
a58ff9cb20 | ||
|
|
d13171cdde | ||
|
|
c1c08c472f | ||
|
|
a51ae7a81a | ||
|
|
3b1b9a8c4d | ||
|
|
027ec57aee | ||
|
|
64e307cc29 | ||
|
|
2663f2edb5 | ||
|
|
c266ac5459 | ||
|
|
95fecb91c8 | ||
|
|
b1c8ac7de6 | ||
|
|
bf150e442e | ||
|
|
ed13dc8c6c | ||
|
|
c7d31883bc | ||
|
|
83afb81ac7 | ||
|
|
2d9ce44329 | ||
|
|
f4261f34e9 | ||
|
|
429658d4cc | ||
|
|
e0833da5dc |
No files matched your search
@@ -15,7 +15,7 @@ jobs:
|
||||
uses: astral-sh/setup-uv@v5
|
||||
with:
|
||||
version: "0.5.14"
|
||||
python-version: "3.9"
|
||||
python-version: "3.10"
|
||||
|
||||
- name: Check if lockfile is up to date
|
||||
run: uv lock --check
|
||||
|
||||
@@ -12,7 +12,7 @@ jobs:
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
matrix:
|
||||
python-version: ["3.9", "3.10", "3.11", "3.12"]
|
||||
python-version: ["3.10", "3.11", "3.12", "3.13"]
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
3.10
|
||||
@@ -0,0 +1,76 @@
|
||||
# Troubleshooting cuDNN Loading Errors
|
||||
|
||||
This guide helps resolve common cuDNN-related errors when running WhisperX on GPU. These issues typically occur when the system can't locate cuDNN libraries or finds conflicting versions.
|
||||
|
||||
## Unable to Load cuDNN Libraries
|
||||
|
||||
If you encounter the following error when running WhisperX:
|
||||
|
||||
`Unable to load any of {libcudnn_cnn.so.9.1.0, libcudnn_cnn.so.9.1, libcudnn_cnn.so.9, libcudnn_cnn.so}`
|
||||
|
||||
This means the cuDNN libraries are installed (via whisperx dependencies) but aren't in a location where the system's dynamic linker can find them.
|
||||
|
||||
### Solution 1: Add to LD_LIBRARY_PATH (Recommended)
|
||||
|
||||
Add this at the start of your Python script or notebook:
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
# Get current LD_LIBRARY_PATH
|
||||
original = os.environ.get("LD_LIBRARY_PATH", "")
|
||||
|
||||
cudnn_path = "/usr/local/lib/python3.12/dist-packages/nvidia/cudnn/lib/"
|
||||
os.environ['LD_LIBRARY_PATH'] = original + ":" + cudnn_path
|
||||
```
|
||||
|
||||
**Note:** Adjust the Python version (`python3.12`) to match your environment.
|
||||
|
||||
### Solution 2: Symlink to LD_LIBRARY_PATH Directory
|
||||
|
||||
If Solution 1 didn't work and you still get the "unable to load" error, symlink the libraries to a directory that's already in your `LD_LIBRARY_PATH`:
|
||||
|
||||
1. Check what's in your LD_LIBRARY_PATH: `echo "$LD_LIBRARY_PATH"`
|
||||
2. Assuming that there is only one path set.
|
||||
Symlink the downloaded libcudnn files to that path:
|
||||
`ln -s /usr/local/lib/python3.12/dist-packages/nvidia/cudnn/lib/libcudnn* "$LD_LIBRARY_PATH"/`
|
||||
|
||||
**Note:** If `LD_LIBRARY_PATH` contains multiple paths (separated by `:`), pick one directory and use it instead of `"$LD_LIBRARY_PATH"`. For example: `/usr/lib/x86_64-linux-gnu/`
|
||||
|
||||
## cuDNN Version Incompatibility
|
||||
|
||||
If you encounter this error:
|
||||
|
||||
```
|
||||
RuntimeError: cuDNN version incompatibility: PyTorch was compiled against (9, 10, 2) but found runtime version (9, 2, 1)
|
||||
```
|
||||
|
||||
This means PyTorch is finding a different cuDNN version than the one it was compiled with. **PyTorch comes bundled with its own cuDNN**, but a conflicting cuDNN in `LD_LIBRARY_PATH` is taking precedence.
|
||||
|
||||
### Solution: Remove Conflicting cuDNN from Path
|
||||
|
||||
Check if there's a conflicting cuDNN path:
|
||||
|
||||
```bash
|
||||
echo $LD_LIBRARY_PATH
|
||||
```
|
||||
|
||||
If you see paths pointing to older cuDNN installations (e.g., system-installed cuDNN or manually downloaded), try one of these:
|
||||
|
||||
**Option 1: Clear LD_LIBRARY_PATH temporarily**
|
||||
|
||||
```python
|
||||
import os
|
||||
# Let PyTorch use its bundled cuDNN
|
||||
os.environ.pop('LD_LIBRARY_PATH', None)
|
||||
```
|
||||
|
||||
**Option 2: Set LD_LIBRARY_PATH to only the correct version**
|
||||
|
||||
```python
|
||||
import os
|
||||
# Point only to the cuDNN that matches PyTorch's compiled version
|
||||
os.environ['LD_LIBRARY_PATH'] = "/usr/local/lib/python3.12/dist-packages/nvidia/cudnn/lib/"
|
||||
```
|
||||
|
||||
**Note:** This error is unlikely on a clean install. If it occurs anyway, [open an issue](https://github.com/m-bain/whisperX/issues). If you've modified system libraries or CUDA/cuDNN, the options above should help resolve most cases.
|
||||
@@ -1,5 +1,10 @@
|
||||
<h1 align="center">WhisperX</h1>
|
||||
|
||||
## Recall.ai - Meeting Transcription API
|
||||
|
||||
If you’re looking for a transcription API for meetings, consider checking out [Recall.ai's Meeting Transcription API](https://www.recall.ai/product/meeting-transcription-api?utm_source=github&utm_medium=sponsorship&utm_campaign=mbain-whisperx), an API that works with Zoom, Google Meet, Microsoft Teams, and more. Recall.ai diarizes by pulling the speaker data and separate audio streams from the meeting platforms, which means 100% accurate speaker diarization with actual speaker names.
|
||||
|
||||
|
||||
<p align="center">
|
||||
<a href="https://github.com/m-bain/whisperX/stargazers">
|
||||
<img src="https://img.shields.io/github/stars/m-bain/whisperX.svg?colorA=orange&colorB=orange&logo=github"
|
||||
@@ -57,6 +62,15 @@ This repository provides fast automatic speech recognition (70x realtime with la
|
||||
|
||||
<h2 align="left" id="setup">Setup ⚙️</h2>
|
||||
|
||||
### 0. CUDA Installation
|
||||
|
||||
To use WhisperX with GPU acceleration, install the CUDA toolkit 12.8 before WhisperX. Skip this step if using only the CPU.
|
||||
|
||||
- For **Linux** users, install the CUDA toolkit 12.8 following this guide:
|
||||
[CUDA Installation Guide for Linux](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/).
|
||||
- For **Windows** users, download and install the CUDA toolkit 12.8:
|
||||
[CUDA Downloads](https://developer.nvidia.com/cuda-12-8-1-download-archive).
|
||||
|
||||
### 1. Simple Installation (Recommended)
|
||||
|
||||
The easiest way to install WhisperX is through PyPi:
|
||||
@@ -97,25 +111,6 @@ uv sync --all-extras --dev
|
||||
|
||||
You may also need to install ffmpeg, rust etc. Follow openAI instructions here https://github.com/openai/whisper#setup.
|
||||
|
||||
### Common Issues & Troubleshooting 🔧
|
||||
|
||||
#### libcudnn Dependencies (GPU Users)
|
||||
|
||||
If you're using WhisperX with GPU support and encounter errors like:
|
||||
|
||||
- `Could not load library libcudnn_ops_infer.so.8`
|
||||
- `Unable to load any of {libcudnn_cnn.so.9.1.0, libcudnn_cnn.so.9.1, libcudnn_cnn.so.9, libcudnn_cnn.so}`
|
||||
- `libcudnn_ops_infer.so.8: cannot open shared object file: No such file or directory`
|
||||
|
||||
This means your system is missing the CUDA Deep Neural Network library (cuDNN). This library is needed for GPU acceleration but isn't always installed by default.
|
||||
|
||||
**Install cuDNN (example for apt based systems):**
|
||||
|
||||
```bash
|
||||
sudo apt update
|
||||
sudo apt install libcudnn8 libcudnn8-dev -y
|
||||
```
|
||||
|
||||
### Speaker Diarization
|
||||
|
||||
To **enable Speaker Diarization**, include your Hugging Face access token (read) that you can generate from [Here](https://huggingface.co/settings/tokens) after the `--hf_token` argument and accept the user agreement for the following models: [Segmentation](https://huggingface.co/pyannote/segmentation-3.0) and [Speaker-Diarization-3.1](https://huggingface.co/pyannote/speaker-diarization-3.1) (if you choose to use Speaker-Diarization 2.x, follow requirements [here](https://huggingface.co/pyannote/speaker-diarization) instead.)
|
||||
@@ -149,7 +144,7 @@ To label the transcript with speaker ID's (set number of speakers if known e.g.
|
||||
|
||||
To run on CPU instead of GPU (and for running on Mac OS X):
|
||||
|
||||
whisperx path/to/audio.wav --compute_type int8
|
||||
whisperx path/to/audio.wav --compute_type int8 --device cpu
|
||||
|
||||
### Other languages
|
||||
|
||||
@@ -171,6 +166,7 @@ See more examples in other languages [here](EXAMPLES.md).
|
||||
```python
|
||||
import whisperx
|
||||
import gc
|
||||
from whisperx.diarize import DiarizationPipeline
|
||||
|
||||
device = "cuda"
|
||||
audio_file = "audio.mp3"
|
||||
@@ -201,7 +197,7 @@ print(result["segments"]) # after alignment
|
||||
# import gc; import torch; gc.collect(); torch.cuda.empty_cache(); del model_a
|
||||
|
||||
# 3. Assign speaker labels
|
||||
diarize_model = whisperx.diarize.DiarizationPipeline(use_auth_token=YOUR_HF_TOKEN, device=device)
|
||||
diarize_model = DiarizationPipeline(use_auth_token=YOUR_HF_TOKEN, device=device)
|
||||
|
||||
# add min/max number of speakers if known
|
||||
diarize_segments = diarize_model(audio)
|
||||
|
||||
+34
-8
@@ -2,23 +2,24 @@
|
||||
urls = { repository = "https://github.com/m-bain/whisperx" }
|
||||
authors = [{ name = "Max Bain" }]
|
||||
name = "whisperx"
|
||||
version = "3.4.1"
|
||||
version = "3.7.6"
|
||||
description = "Time-Accurate Automatic Speech Recognition using Whisper."
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.9, <3.13"
|
||||
requires-python = ">=3.10, <3.14"
|
||||
license = { text = "BSD-2-Clause" }
|
||||
|
||||
dependencies = [
|
||||
"ctranslate2<4.5.0",
|
||||
"ctranslate2>=4.5.0",
|
||||
"faster-whisper>=1.1.1",
|
||||
"nltk>=3.9.1",
|
||||
"numpy>=2.0.2",
|
||||
"onnxruntime>=1.19",
|
||||
"numpy>=2.1.0",
|
||||
"pandas>=2.2.3",
|
||||
"pyannote-audio>=3.3.2",
|
||||
"torch>=2.5.1",
|
||||
"torchaudio>=2.5.1",
|
||||
"pyannote-audio>=3.3.2,<4.0.0",
|
||||
"huggingface-hub<1.0.0",
|
||||
"torch~=2.8.0",
|
||||
"torchaudio~=2.8.0",
|
||||
"transformers>=4.48.0",
|
||||
"triton>=3.3.0; sys_platform == 'linux' and platform_machine == 'x86_64'" # only install triton on x86_64 Linux
|
||||
]
|
||||
|
||||
|
||||
@@ -34,3 +35,28 @@ include-package-data = true
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["."]
|
||||
include = ["whisperx*"]
|
||||
|
||||
[tool.uv.sources]
|
||||
torch = [
|
||||
{ index = "pytorch-cpu", marker = "sys_platform == 'darwin'" },
|
||||
{ index = "pytorch-cpu", marker = "platform_machine != 'x86_64' and sys_platform != 'darwin'" },
|
||||
{ index = "pytorch", marker = "platform_machine == 'x86_64' and sys_platform != 'darwin'" },
|
||||
]
|
||||
torchaudio = [
|
||||
{ index = "pytorch-cpu", marker = "sys_platform == 'darwin'" },
|
||||
{ index = "pytorch-cpu", marker = "platform_machine != 'x86_64' and sys_platform != 'darwin'" },
|
||||
{ index = "pytorch", marker = "platform_machine == 'x86_64' and sys_platform != 'darwin'" },
|
||||
]
|
||||
triton = [
|
||||
{ index = "pytorch", marker = "sys_platform == 'linux'" },
|
||||
]
|
||||
|
||||
[[tool.uv.index]]
|
||||
name = "pytorch"
|
||||
url = "https://download.pytorch.org/whl/cu128"
|
||||
explicit = true
|
||||
|
||||
[[tool.uv.index]]
|
||||
name = "pytorch-cpu"
|
||||
url = "https://download.pytorch.org/whl/cpu"
|
||||
explicit = true
|
||||
@@ -29,3 +29,29 @@ def load_audio(*args, **kwargs):
|
||||
def assign_word_speakers(*args, **kwargs):
|
||||
diarize = _lazy_import("diarize")
|
||||
return diarize.assign_word_speakers(*args, **kwargs)
|
||||
|
||||
|
||||
def setup_logging(*args, **kwargs):
|
||||
"""
|
||||
Configure logging for WhisperX.
|
||||
|
||||
Args:
|
||||
level: Logging level (debug, info, warning, error, critical). Default: warning
|
||||
log_file: Optional path to log file. If None, logs only to console.
|
||||
"""
|
||||
logging_module = _lazy_import("log_utils")
|
||||
return logging_module.setup_logging(*args, **kwargs)
|
||||
|
||||
|
||||
def get_logger(*args, **kwargs):
|
||||
"""
|
||||
Get a logger instance for the given module.
|
||||
|
||||
Args:
|
||||
name: Logger name (typically __name__ from calling module)
|
||||
|
||||
Returns:
|
||||
Logger instance configured with WhisperX settings
|
||||
"""
|
||||
logging_module = _lazy_import("log_utils")
|
||||
return logging_module.get_logger(*args, **kwargs)
|
||||
@@ -6,6 +6,7 @@ import torch
|
||||
|
||||
from whisperx.utils import (LANGUAGES, TO_LANGUAGE_CODE, optional_float,
|
||||
optional_int, str2bool)
|
||||
from whisperx.log_utils import setup_logging
|
||||
|
||||
|
||||
def cli():
|
||||
@@ -23,6 +24,7 @@ def cli():
|
||||
parser.add_argument("--output_dir", "-o", type=str, default=".", help="directory to save the outputs")
|
||||
parser.add_argument("--output_format", "-f", type=str, default="all", choices=["all", "srt", "vtt", "txt", "tsv", "json", "aud"], help="format of the output file; if not specified, all available formats will be produced")
|
||||
parser.add_argument("--verbose", type=str2bool, default=True, help="whether to print out the progress and debug messages")
|
||||
parser.add_argument("--log-level", type=str, default=None, choices=["debug", "info", "warning", "error", "critical"], help="logging level (overrides --verbose if set)")
|
||||
|
||||
parser.add_argument("--task", type=str, default="transcribe", choices=["transcribe", "translate"], help="whether to perform X->X speech recognition ('transcribe') or X->English translation ('translate')")
|
||||
parser.add_argument("--language", type=str, default=None, choices=sorted(LANGUAGES.keys()) + sorted([k.title() for k in TO_LANGUAGE_CODE.keys()]), help="language spoken in the audio, specify None to perform language detection")
|
||||
@@ -56,6 +58,7 @@ def cli():
|
||||
parser.add_argument("--suppress_numerals", action="store_true", help="whether to suppress numeric symbols and currency symbols during sampling, since wav2vec2 cannot align them correctly")
|
||||
|
||||
parser.add_argument("--initial_prompt", type=str, default=None, help="optional text to provide as a prompt for the first window.")
|
||||
parser.add_argument("--hotwords", type=str, default=None, help="hotwords/hint phrases to the model (e.g. \"WhisperX, PyAnnote, GPU\"); improves recognition of rare/technical terms")
|
||||
parser.add_argument("--condition_on_previous_text", type=str2bool, default=False, help="if True, provide the previous output of the model as a prompt for the next window; disabling may make the text inconsistent across windows, but the model becomes less prone to getting stuck in a failure loop")
|
||||
parser.add_argument("--fp16", type=str2bool, default=True, help="whether to perform inference in fp16; True by default")
|
||||
|
||||
@@ -80,6 +83,16 @@ def cli():
|
||||
|
||||
args = parser.parse_args().__dict__
|
||||
|
||||
log_level = args.get("log_level")
|
||||
verbose = args.get("verbose")
|
||||
|
||||
if log_level is not None:
|
||||
setup_logging(level=log_level)
|
||||
elif verbose:
|
||||
setup_logging(level="info")
|
||||
else:
|
||||
setup_logging(level="warning")
|
||||
|
||||
from whisperx.transcribe import transcribe_task
|
||||
|
||||
transcribe_task(args, parser)
|
||||
|
||||
+34
-26
@@ -14,17 +14,19 @@ import torchaudio
|
||||
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
|
||||
|
||||
from whisperx.audio import SAMPLE_RATE, load_audio
|
||||
from whisperx.utils import interpolate_nans
|
||||
from whisperx.types import (
|
||||
from whisperx.utils import interpolate_nans, PUNKT_LANGUAGES
|
||||
from whisperx.schema import (
|
||||
AlignedTranscriptionResult,
|
||||
SingleSegment,
|
||||
SingleAlignedSegment,
|
||||
SingleWordSegment,
|
||||
SegmentData,
|
||||
)
|
||||
from nltk.tokenize.punkt import PunktSentenceTokenizer, PunktParameters
|
||||
import nltk
|
||||
from nltk.data import load as nltk_load
|
||||
from whisperx.log_utils import get_logger
|
||||
|
||||
PUNKT_ABBREVIATIONS = ['dr', 'vs', 'mr', 'mrs', 'prof']
|
||||
logger = get_logger(__name__)
|
||||
|
||||
LANGUAGES_WITHOUT_SPACES = ["ja", "zh"]
|
||||
|
||||
@@ -53,7 +55,7 @@ DEFAULT_ALIGN_MODELS_HF = {
|
||||
"tr": "mpoyraz/wav2vec2-xls-r-300m-cv7-turkish",
|
||||
"da": "saattrupdan/wav2vec2-xls-r-300m-ftspeech",
|
||||
"he": "imvladikon/wav2vec2-xls-r-300m-hebrew",
|
||||
"vi": 'nguyenvulebinh/wav2vec2-base-vi',
|
||||
"vi": 'nguyenvulebinh/wav2vec2-base-vi-vlsp2020',
|
||||
"ko": "kresnik/wav2vec2-large-xlsr-korean",
|
||||
"ur": "kingabzpro/wav2vec2-large-xls-r-300m-Urdu",
|
||||
"te": "anuragshas/wav2vec2-large-xlsr-53-telugu",
|
||||
@@ -71,6 +73,7 @@ DEFAULT_ALIGN_MODELS_HF = {
|
||||
"ka": "xsway/wav2vec2-large-xlsr-georgian",
|
||||
"lv": "jimregan/wav2vec2-large-xlsr-latvian-cv",
|
||||
"tl": "Khalsuu/filipino-wav2vec2-l-xls-r-300m-official",
|
||||
"sv": "KBLab/wav2vec2-large-voxrex-swedish",
|
||||
}
|
||||
|
||||
|
||||
@@ -82,8 +85,9 @@ def load_align_model(language_code: str, device: str, model_name: Optional[str]
|
||||
elif language_code in DEFAULT_ALIGN_MODELS_HF:
|
||||
model_name = DEFAULT_ALIGN_MODELS_HF[language_code]
|
||||
else:
|
||||
print(f"There is no default alignment model set for this language ({language_code}).\
|
||||
Please find a wav2vec2.0 model finetuned on this language in https://huggingface.co/models, then pass the model name in --align_model [MODEL_NAME]")
|
||||
logger.error(f"No default alignment model for language: {language_code}. "
|
||||
f"Please find a wav2vec2.0 model finetuned on this language at https://huggingface.co/models, "
|
||||
f"then pass the model name via --align_model [MODEL_NAME]")
|
||||
raise ValueError(f"No default align-model for language: {language_code}")
|
||||
|
||||
if model_name in torchaudio.pipelines.__all__:
|
||||
@@ -124,14 +128,14 @@ def align(
|
||||
"""
|
||||
Align phoneme recognition predictions to known transcription.
|
||||
"""
|
||||
|
||||
|
||||
if not torch.is_tensor(audio):
|
||||
if isinstance(audio, str):
|
||||
audio = load_audio(audio)
|
||||
audio = torch.from_numpy(audio)
|
||||
if len(audio.shape) == 1:
|
||||
audio = audio.unsqueeze(0)
|
||||
|
||||
|
||||
MAX_DURATION = audio.shape[1] / SAMPLE_RATE
|
||||
|
||||
model_dictionary = align_model_metadata["dictionary"]
|
||||
@@ -148,7 +152,7 @@ def align(
|
||||
base_progress = ((sdx + 1) / total_segments) * 100
|
||||
percent_complete = (50 + base_progress / 2) if combined_progress else base_progress
|
||||
print(f"Progress: {percent_complete:.2f}%...")
|
||||
|
||||
|
||||
num_leading = len(segment["text"]) - len(segment["text"].lstrip())
|
||||
num_trailing = len(segment["text"]) - len(segment["text"].rstrip())
|
||||
text = segment["text"]
|
||||
@@ -165,7 +169,7 @@ def align(
|
||||
# wav2vec2 models use "|" character to represent spaces
|
||||
if model_lang not in LANGUAGES_WITHOUT_SPACES:
|
||||
char_ = char_.replace(" ", "|")
|
||||
|
||||
|
||||
# ignore whitespace at beginning and end of transcript
|
||||
if cdx < num_leading:
|
||||
pass
|
||||
@@ -187,10 +191,14 @@ def align(
|
||||
# index for placeholder
|
||||
clean_wdx.append(wdx)
|
||||
|
||||
|
||||
punkt_param = PunktParameters()
|
||||
punkt_param.abbrev_types = set(PUNKT_ABBREVIATIONS)
|
||||
sentence_splitter = PunktSentenceTokenizer(punkt_param)
|
||||
|
||||
# Use language-specific Punkt model if available otherwise we fallback to English.
|
||||
punkt_lang = PUNKT_LANGUAGES.get(model_lang, 'english')
|
||||
try:
|
||||
sentence_splitter = nltk_load(f'tokenizers/punkt_tab/{punkt_lang}.pickle')
|
||||
except LookupError:
|
||||
nltk.download('punkt_tab', quiet=True)
|
||||
sentence_splitter = nltk_load(f'tokenizers/punkt_tab/{punkt_lang}.pickle')
|
||||
sentence_spans = list(sentence_splitter.span_tokenize(text))
|
||||
|
||||
segment_data[sdx] = {
|
||||
@@ -199,12 +207,12 @@ def align(
|
||||
"clean_wdx": clean_wdx,
|
||||
"sentence_spans": sentence_spans
|
||||
}
|
||||
|
||||
|
||||
aligned_segments: List[SingleAlignedSegment] = []
|
||||
|
||||
|
||||
# 2. Get prediction matrix from alignment model & align
|
||||
for sdx, segment in enumerate(transcript):
|
||||
|
||||
|
||||
t1 = segment["start"]
|
||||
t2 = segment["end"]
|
||||
text = segment["text"]
|
||||
@@ -222,12 +230,12 @@ def align(
|
||||
|
||||
# check we can align
|
||||
if len(segment_data[sdx]["clean_char"]) == 0:
|
||||
print(f'Failed to align segment ("{segment["text"]}"): no characters in this segment found in model dictionary, resorting to original...')
|
||||
logger.warning(f'Failed to align segment ("{segment["text"]}"): no characters in this segment found in model dictionary, resorting to original')
|
||||
aligned_segments.append(aligned_seg)
|
||||
continue
|
||||
|
||||
if t1 >= MAX_DURATION:
|
||||
print(f'Failed to align segment ("{segment["text"]}"): original start time longer than audio duration, skipping...')
|
||||
logger.warning(f'Failed to align segment ("{segment["text"]}"): original start time longer than audio duration, skipping')
|
||||
aligned_segments.append(aligned_seg)
|
||||
continue
|
||||
|
||||
@@ -247,7 +255,7 @@ def align(
|
||||
)
|
||||
else:
|
||||
lengths = None
|
||||
|
||||
|
||||
with torch.inference_mode():
|
||||
if model_type == "torchaudio":
|
||||
emissions, _ = model(waveform_segment.to(device), lengths=lengths)
|
||||
@@ -269,7 +277,7 @@ def align(
|
||||
path = backtrack_beam(trellis, emission, tokens, blank_id, beam_width=2)
|
||||
|
||||
if path is None:
|
||||
print(f'Failed to align segment ("{segment["text"]}"): backtrack failed, resorting to original...')
|
||||
logger.warning(f'Failed to align segment ("{segment["text"]}"): backtrack failed, resorting to original')
|
||||
aligned_segments.append(aligned_seg)
|
||||
continue
|
||||
|
||||
@@ -304,7 +312,7 @@ def align(
|
||||
word_idx += 1
|
||||
elif cdx == len(text) - 1 or text[cdx+1] == " ":
|
||||
word_idx += 1
|
||||
|
||||
|
||||
char_segments_arr = pd.DataFrame(char_segments_arr)
|
||||
|
||||
aligned_subsegments = []
|
||||
@@ -333,7 +341,7 @@ def align(
|
||||
word_end = word_chars["end"].max()
|
||||
word_score = round(word_chars["score"].mean(), 3)
|
||||
|
||||
# -1 indicates unalignable
|
||||
# -1 indicates unalignable
|
||||
word_segment = {"word": word_text}
|
||||
|
||||
if not np.isnan(word_start):
|
||||
@@ -344,7 +352,7 @@ def align(
|
||||
word_segment["score"] = word_score
|
||||
|
||||
sentence_words.append(word_segment)
|
||||
|
||||
|
||||
aligned_subsegments.append({
|
||||
"text": sentence_text,
|
||||
"start": sentence_start,
|
||||
@@ -424,7 +432,7 @@ def get_wildcard_emission(frame_emission, tokens, blank_id):
|
||||
wildcard_mask = (tokens == -1)
|
||||
|
||||
# Get scores for non-wildcard positions
|
||||
regular_scores = frame_emission[tokens.clamp(min=0)] # clamp to avoid -1 index
|
||||
regular_scores = frame_emission[tokens.clamp(min=0).long()] # clamp to avoid -1 index
|
||||
|
||||
# Create a mask and compute the maximum value without modifying frame_emission
|
||||
max_valid_score = frame_emission.clone() # Create a copy
|
||||
|
||||
+15
-7
@@ -12,8 +12,11 @@ from transformers import Pipeline
|
||||
from transformers.pipelines.pt_utils import PipelineIterator
|
||||
|
||||
from whisperx.audio import N_SAMPLES, SAMPLE_RATE, load_audio, log_mel_spectrogram
|
||||
from whisperx.types import SingleSegment, TranscriptionResult
|
||||
from whisperx.schema import SingleSegment, TranscriptionResult
|
||||
from whisperx.vads import Vad, Silero, Pyannote
|
||||
from whisperx.log_utils import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
def find_numeral_symbol_tokens(tokenizer):
|
||||
@@ -247,7 +250,7 @@ class FasterWhisperPipeline(Pipeline):
|
||||
if self.suppress_numerals:
|
||||
previous_suppress_tokens = self.options.suppress_tokens
|
||||
numeral_symbol_tokens = find_numeral_symbol_tokens(self.tokenizer)
|
||||
print(f"Suppressing numeral and symbol tokens")
|
||||
logger.info("Suppressing numeral and symbol tokens")
|
||||
new_suppressed_tokens = numeral_symbol_tokens + self.options.suppress_tokens
|
||||
new_suppressed_tokens = list(set(new_suppressed_tokens))
|
||||
self.options = replace(self.options, suppress_tokens=new_suppressed_tokens)
|
||||
@@ -285,7 +288,7 @@ class FasterWhisperPipeline(Pipeline):
|
||||
|
||||
def detect_language(self, audio: np.ndarray) -> str:
|
||||
if audio.shape[0] < N_SAMPLES:
|
||||
print("Warning: audio is shorter than 30s, language detection may be inaccurate.")
|
||||
logger.warning("Audio is shorter than 30s, language detection may be inaccurate")
|
||||
model_n_mels = self.model.feat_kwargs.get("feature_size")
|
||||
segment = log_mel_spectrogram(audio[: N_SAMPLES],
|
||||
n_mels=model_n_mels if model_n_mels is not None else 80,
|
||||
@@ -294,7 +297,7 @@ class FasterWhisperPipeline(Pipeline):
|
||||
results = self.model.model.detect_language(encoder_output)
|
||||
language_token, language_probability = results[0][0]
|
||||
language = language_token[2:-2]
|
||||
print(f"Detected language: {language} ({language_probability:.2f}) in first 30s of audio...")
|
||||
logger.info(f"Detected language: {language} ({language_probability:.2f}) in first 30s of audio")
|
||||
return language
|
||||
|
||||
|
||||
@@ -319,7 +322,8 @@ def load_model(
|
||||
whisper_arch - The name of the Whisper model to load.
|
||||
device - The device to load the model on.
|
||||
compute_type - The compute type to use for the model.
|
||||
vad_method - The vad method to use. vad_model has higher priority if is not None.
|
||||
vad_model - The vad model to manually assign.
|
||||
vad_method - The vad method to use. vad_model has a higher priority if it is not None.
|
||||
options - A dictionary of options to use for the model.
|
||||
language - The language of the model. (use English for now)
|
||||
model - The WhisperModel instance to use.
|
||||
@@ -343,7 +347,7 @@ def load_model(
|
||||
if language is not None:
|
||||
tokenizer = Tokenizer(model.hf_tokenizer, model.model.is_multilingual, task=task, language=language)
|
||||
else:
|
||||
print("No language specified, language will be first be detected for each audio file (increases inference time).")
|
||||
logger.info("No language specified, language will be detected for each audio file (increases inference time)")
|
||||
tokenizer = None
|
||||
|
||||
default_asr_options = {
|
||||
@@ -401,7 +405,11 @@ def load_model(
|
||||
if vad_method == "silero":
|
||||
vad_model = Silero(**default_vad_options)
|
||||
elif vad_method == "pyannote":
|
||||
vad_model = Pyannote(torch.device(device), use_auth_token=None, **default_vad_options)
|
||||
if device == 'cuda':
|
||||
device_vad = f'cuda:{device_index}'
|
||||
else:
|
||||
device_vad = device
|
||||
vad_model = Pyannote(torch.device(device_vad), use_auth_token=None, **default_vad_options)
|
||||
else:
|
||||
raise ValueError(f"Invalid vad_method: {vad_method}")
|
||||
|
||||
|
||||
+5
-1
@@ -5,7 +5,10 @@ from typing import Optional, Union
|
||||
import torch
|
||||
|
||||
from whisperx.audio import load_audio, SAMPLE_RATE
|
||||
from whisperx.types import TranscriptionResult, AlignedTranscriptionResult
|
||||
from whisperx.schema import TranscriptionResult, AlignedTranscriptionResult
|
||||
from whisperx.log_utils import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class DiarizationPipeline:
|
||||
@@ -18,6 +21,7 @@ class DiarizationPipeline:
|
||||
if isinstance(device, str):
|
||||
device = torch.device(device)
|
||||
model_config = model_name or "pyannote/speaker-diarization-3.1"
|
||||
logger.info(f"Loading diarization model: {model_config}")
|
||||
self.model = Pipeline.from_pretrained(model_config, use_auth_token=use_auth_token).to(device)
|
||||
|
||||
def __call__(
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
import logging
|
||||
import sys
|
||||
from typing import Optional
|
||||
|
||||
_LOG_FORMAT = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
|
||||
_DATE_FORMAT = "%Y-%m-%d %H:%M:%S"
|
||||
|
||||
|
||||
def setup_logging(
|
||||
level: str = "info",
|
||||
log_file: Optional[str] = None,
|
||||
) -> None:
|
||||
"""
|
||||
Configure logging for WhisperX.
|
||||
|
||||
Args:
|
||||
level: Logging level (debug, info, warning, error, critical). Default: info
|
||||
log_file: Optional path to log file. If None, logs only to console.
|
||||
"""
|
||||
logger = logging.getLogger("whisperx")
|
||||
|
||||
logger.handlers.clear()
|
||||
|
||||
try:
|
||||
log_level = getattr(logging, level.upper())
|
||||
except AttributeError:
|
||||
log_level = logging.WARNING
|
||||
logger.setLevel(log_level)
|
||||
|
||||
formatter = logging.Formatter(_LOG_FORMAT, datefmt=_DATE_FORMAT)
|
||||
|
||||
console_handler = logging.StreamHandler(sys.stdout)
|
||||
console_handler.setLevel(log_level)
|
||||
console_handler.setFormatter(formatter)
|
||||
|
||||
logger.addHandler(console_handler)
|
||||
|
||||
if log_file:
|
||||
try:
|
||||
file_handler = logging.FileHandler(log_file)
|
||||
file_handler.setLevel(log_level)
|
||||
file_handler.setFormatter(formatter)
|
||||
logger.addHandler(file_handler)
|
||||
except (OSError) as e:
|
||||
logger.warning(f"Failed to create log file '{log_file}': {e}")
|
||||
logger.warning("Continuing with console logging only")
|
||||
|
||||
# Don't propagate to root logger to avoid duplicate messages
|
||||
logger.propagate = False
|
||||
|
||||
|
||||
def get_logger(name: str) -> logging.Logger:
|
||||
"""
|
||||
Get a logger instance for the given module.
|
||||
|
||||
Args:
|
||||
name: Logger name (typically __name__ from calling module)
|
||||
|
||||
Returns:
|
||||
Logger instance configured with WhisperX settings
|
||||
"""
|
||||
whisperx_logger = logging.getLogger("whisperx")
|
||||
if not whisperx_logger.handlers:
|
||||
setup_logging()
|
||||
|
||||
logger_name = "whisperx" if name == "__main__" else name
|
||||
return logging.getLogger(logger_name)
|
||||
File renamed without changes.
+12
-9
@@ -10,8 +10,11 @@ from whisperx.alignment import align, load_align_model
|
||||
from whisperx.asr import load_model
|
||||
from whisperx.audio import load_audio
|
||||
from whisperx.diarize import DiarizationPipeline, assign_word_speakers
|
||||
from whisperx.types import AlignedTranscriptionResult, TranscriptionResult
|
||||
from whisperx.schema import AlignedTranscriptionResult, TranscriptionResult
|
||||
from whisperx.utils import LANGUAGES, TO_LANGUAGE_CODE, get_writer
|
||||
from whisperx.log_utils import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
def transcribe_task(args: dict, parser: argparse.ArgumentParser):
|
||||
@@ -103,6 +106,7 @@ def transcribe_task(args: dict, parser: argparse.ArgumentParser):
|
||||
"no_speech_threshold": args.pop("no_speech_threshold"),
|
||||
"condition_on_previous_text": False,
|
||||
"initial_prompt": args.pop("initial_prompt"),
|
||||
"hotwords": args.pop("hotwords"),
|
||||
"suppress_tokens": [int(x) for x in args.pop("suppress_tokens").split(",")],
|
||||
"suppress_numerals": args.pop("suppress_numerals"),
|
||||
}
|
||||
@@ -119,7 +123,6 @@ def transcribe_task(args: dict, parser: argparse.ArgumentParser):
|
||||
|
||||
# Part 1: VAD & ASR Loop
|
||||
results = []
|
||||
tmp_results = []
|
||||
# model = load_model(model_name, device=device, download_root=model_dir)
|
||||
model = load_model(
|
||||
model_name,
|
||||
@@ -143,7 +146,7 @@ def transcribe_task(args: dict, parser: argparse.ArgumentParser):
|
||||
for audio_path in args.pop("audio"):
|
||||
audio = load_audio(audio_path)
|
||||
# >> VAD & ASR
|
||||
print(">>Performing transcription...")
|
||||
logger.info("Performing transcription...")
|
||||
result: TranscriptionResult = model.transcribe(
|
||||
audio,
|
||||
batch_size=batch_size,
|
||||
@@ -176,13 +179,13 @@ def transcribe_task(args: dict, parser: argparse.ArgumentParser):
|
||||
if align_model is not None and len(result["segments"]) > 0:
|
||||
if result.get("language", "en") != align_metadata["language"]:
|
||||
# load new language
|
||||
print(
|
||||
logger.info(
|
||||
f"New language found ({result['language']})! Previous was ({align_metadata['language']}), loading new alignment model for new language..."
|
||||
)
|
||||
align_model, align_metadata = load_align_model(
|
||||
result["language"], device
|
||||
)
|
||||
print(">>Performing alignment...")
|
||||
logger.info("Performing alignment...")
|
||||
result: AlignedTranscriptionResult = align(
|
||||
result["segments"],
|
||||
align_model,
|
||||
@@ -204,12 +207,12 @@ def transcribe_task(args: dict, parser: argparse.ArgumentParser):
|
||||
# >> Diarize
|
||||
if diarize:
|
||||
if hf_token is None:
|
||||
print(
|
||||
"Warning, no --hf_token used, needs to be saved in environment variable, otherwise will throw error loading diarization model..."
|
||||
logger.warning(
|
||||
"No --hf_token provided, needs to be saved in environment variable, otherwise will throw error loading diarization model"
|
||||
)
|
||||
tmp_results = results
|
||||
print(">>Performing diarization...")
|
||||
print(">>Using model:", diarize_model_name)
|
||||
logger.info("Performing diarization...")
|
||||
logger.info(f"Using model: {diarize_model_name}")
|
||||
results = []
|
||||
diarize_model = DiarizationPipeline(model_name=diarize_model_name, use_auth_token=hf_token, device=device)
|
||||
for result, input_audio_path in tmp_results:
|
||||
|
||||
+25
-2
@@ -126,6 +126,29 @@ TO_LANGUAGE_CODE = {
|
||||
|
||||
LANGUAGES_WITHOUT_SPACES = ["ja", "zh"]
|
||||
|
||||
# Mapping of language codes to NLTK Punkt tokenizer model names
|
||||
PUNKT_LANGUAGES = {
|
||||
'cs': 'czech',
|
||||
'da': 'danish',
|
||||
'de': 'german',
|
||||
'el': 'greek',
|
||||
'en': 'english',
|
||||
'es': 'spanish',
|
||||
'et': 'estonian',
|
||||
'fi': 'finnish',
|
||||
'fr': 'french',
|
||||
'it': 'italian',
|
||||
'nl': 'dutch',
|
||||
'no': 'norwegian',
|
||||
'pl': 'polish',
|
||||
'pt': 'portuguese',
|
||||
'sl': 'slovene',
|
||||
'sv': 'swedish',
|
||||
'tr': 'turkish',
|
||||
"ml": "malayalam",
|
||||
"ru": "russian",
|
||||
}
|
||||
|
||||
system_encoding = sys.getdefaultencoding()
|
||||
|
||||
if system_encoding != "utf-8":
|
||||
@@ -410,7 +433,7 @@ class WriteJSON(ResultWriter):
|
||||
|
||||
def get_writer(
|
||||
output_format: str, output_dir: str
|
||||
) -> Callable[[dict, TextIO, dict], None]:
|
||||
) -> Callable[[dict, str, dict], None]:
|
||||
writers = {
|
||||
"txt": WriteTXT,
|
||||
"vtt": WriteVTT,
|
||||
@@ -425,7 +448,7 @@ def get_writer(
|
||||
if output_format == "all":
|
||||
all_writers = [writer(output_dir) for writer in writers.values()]
|
||||
|
||||
def write_all(result: dict, file: TextIO, options: dict):
|
||||
def write_all(result: dict, file: str, options: dict):
|
||||
for writer in all_writers:
|
||||
writer(result, file, options)
|
||||
|
||||
|
||||
@@ -13,6 +13,9 @@ from pyannote.core import Segment
|
||||
|
||||
from whisperx.diarize import Segment as SegmentX
|
||||
from whisperx.vads.vad import Vad
|
||||
from whisperx.log_utils import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
def load_vad_model(device, vad_onset=0.500, vad_offset=0.363, use_auth_token=None, model_fp=None):
|
||||
@@ -232,7 +235,7 @@ class VoiceActivitySegmentation(VoiceActivityDetection):
|
||||
class Pyannote(Vad):
|
||||
|
||||
def __init__(self, device, use_auth_token=None, model_fp=None, **kwargs):
|
||||
print(">>Performing voice activity detection using Pyannote...")
|
||||
logger.info("Performing voice activity detection using Pyannote...")
|
||||
super().__init__(kwargs['vad_onset'])
|
||||
self.vad_pipeline = load_vad_model(device, use_auth_token=use_auth_token, model_fp=model_fp)
|
||||
|
||||
@@ -257,7 +260,7 @@ class Pyannote(Vad):
|
||||
segments_list.append(SegmentX(speech_turn.start, speech_turn.end, "UNKNOWN"))
|
||||
|
||||
if len(segments_list) == 0:
|
||||
print("No active speech found in audio")
|
||||
logger.warning("No active speech found in audio")
|
||||
return []
|
||||
assert segments_list, "segments_list is empty."
|
||||
return Vad.merge_chunks(segments_list, chunk_size, onset, offset)
|
||||
@@ -8,6 +8,9 @@ import torch
|
||||
|
||||
from whisperx.diarize import Segment as SegmentX
|
||||
from whisperx.vads.vad import Vad
|
||||
from whisperx.log_utils import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
AudioFile = Union[Text, Path, IOBase, Mapping]
|
||||
|
||||
@@ -15,7 +18,7 @@ AudioFile = Union[Text, Path, IOBase, Mapping]
|
||||
class Silero(Vad):
|
||||
# check again default values
|
||||
def __init__(self, **kwargs):
|
||||
print(">>Performing voice activity detection using Silero...")
|
||||
logger.info("Performing voice activity detection using Silero...")
|
||||
super().__init__(kwargs['vad_onset'])
|
||||
|
||||
self.vad_onset = kwargs['vad_onset']
|
||||
@@ -60,7 +63,7 @@ class Silero(Vad):
|
||||
):
|
||||
assert chunk_size > 0
|
||||
if len(segments_list) == 0:
|
||||
print("No active speech found in audio")
|
||||
logger.warning("No active speech found in audio")
|
||||
return []
|
||||
assert segments_list, "segments_list is empty."
|
||||
return Vad.merge_chunks(segments_list, chunk_size, onset, offset)
|
||||
@@ -52,23 +52,3 @@ class Vad:
|
||||
|
||||
return merged_segments
|
||||
|
||||
# Unused function
|
||||
@staticmethod
|
||||
def merge_vad(vad_arr, pad_onset=0.0, pad_offset=0.0, min_duration_off=0.0, min_duration_on=0.0):
|
||||
active = Annotation()
|
||||
for k, vad_t in enumerate(vad_arr):
|
||||
region = Segment(vad_t[0] - pad_onset, vad_t[1] + pad_offset)
|
||||
active[region, k] = 1
|
||||
|
||||
if pad_offset > 0.0 or pad_onset > 0.0 or min_duration_off > 0.0:
|
||||
active = active.support(collar=min_duration_off)
|
||||
|
||||
# remove tracks shorter than min_duration_on
|
||||
if min_duration_on > 0:
|
||||
for segment, track in list(active.itertracks()):
|
||||
if segment.duration < min_duration_on:
|
||||
del active[segment, track]
|
||||
|
||||
active = active.for_json()
|
||||
active_segs = pd.DataFrame([x['segment'] for x in active['content']])
|
||||
return active_segs
|
||||
Reference in new issue
Block a user