First push
This commit is contained in:
seclusionenthusiast committed 2026-03-03 14:52:19 +09:00
commit c4a020c191
146 files changed
+40796

No files matched your search

+201
View File
@@ -0,0 +1,201 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
+47
View File
@@ -0,0 +1,47 @@
# Lyrics Sync (lsync)
This project aims to build a lyrics-to-audio alignment system that can synchronize the audio of a polyphonic song with its lyrics and produce time-aligned lyrics with word-level onset and offset as a `.lrc` file. A deep-learning-based system is developed to approach the problem in three steps, which include separating the vocals, recognizing the singing vocals, and performing forced alignment. For singing vocals recognition, transfer learning is utilized to apply knowledge obtained from the speech domain to the singing domain.
### Flow
![](.github/fig.png)
### Installation
```shell
conda env update -f environment.yml
conda activate lsync
```
### Usage
```python
from lsync import LyricsSync
lsync = LyricsSync()
words, lrc = lsync.sync(audio_path, lyrics_path)
```
#### Demo
Please refer to [demo.ipynb](./demo.ipynb).
If you want to visualize `.lrc` for evaluation, you can use [Lrc Player](https://github.com/mikezzb/lrc-player).
### Experiments
If you want to fine-tune a Wav2Vec2 model for better accuracy in singing domain, please refer to the experiments section below.
#### Fine-tune Wav2Vec2 model for lyrics transcription
1. Make a `dataset` folder in root folder
2. Download [DALI](https://github.com/gabolsgabs/DALI) dataset and put it it inside `dataset/DALI/v1`
* Similarly, you can download jamendolyrics dataset for evaluation and put it in `dataset/jamendolyrics`
3. Download all DALI songs using `python get_dataset.py`
4. Run `dataset.ipynb` to prepare the DALI for fine-tune tasks
* Procedures including vocal extraction, line-level segmentation, and making tokenizer
5. Run `train.ipynb` to fine-tune the `facebook/wav2vec2-base` for singing voice recognition
6. Run `run.ipynb` to see how to use the `lsync` library for lyrics-to-audio alignment based on the fine-tuned model
* Remember to update model path to your model's path inside `lsync/phoneme_recognizer.py`
+367
View File
@@ -0,0 +1,367 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Prepare DALI for fine-tune task"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Extract Vocals"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import DALI as dali_code\n",
"import os\n",
"import torch\n",
"from lsync.voice_extractor import VoiceExtractor\n",
"from lsync.config import ORIGINAL_SR, TARGET_SR\n",
"from lsync.util import save_audio, save_audio_file\n",
"from tqdm.notebook import tqdm\n",
"\n",
"\n",
"path_audio = os.path.abspath('dataset/DALI/audio')\n",
"path_vocals = os.path.abspath(\"dataset/DALI/processed_audio\")\n",
"torch.cuda.is_available()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"converted = set([os.path.splitext(x)[0] for x in os.listdir(path_vocals)])\n",
"ve = VoiceExtractor()\n",
"\n",
"for fname in tqdm(os.listdir(path_audio)):\n",
" try:\n",
" audio_name, ext = os.path.splitext(fname)\n",
" if ext != '.mp3' or audio_name in converted:\n",
" continue\n",
" file_path = os.path.join(path_audio, fname)\n",
" vocals = ve.extract_voice(file_path, post_process=True)\n",
" save_audio(vocals, audio_name, sr=TARGET_SR, out_path=path_vocals)\n",
" except Exception as e:\n",
" print(e)\n",
" continue"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Make Speech Recognition Dataset for Fine-tune"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Load DALI annotation"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import DALI as dali_code\n",
"import os\n",
"from lsync.util import save_audio_file\n",
"import pandas as pd\n",
"import librosa\n",
"\n",
"dali_data_path = os.path.abspath('dataset/DALI/v1')\n",
"path_audio = os.path.abspath('dataset/DALI/audio')\n",
"dali_data = dali_code.get_the_DALI_dataset(dali_data_path, skip=[], keep=[])\n",
"dali_info = dali_code.get_info(dali_data_path + '/info/DALI_DATA_INFO.gz')\n",
"path_segments = os.path.abspath(\"dataset/DALI/segmented_audio\")\n",
"csv_path = os.path.abspath('dali_lines.csv')"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Make lyrics line level dataset"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"processed_songs = [os.path.splitext(x)[0] for x in os.listdir(path_vocals)]\n",
"\n",
"def extract_audio_segment(audio, time_seg, save_path, sr = TARGET_SR):\n",
" start_t, end_t = time_seg\n",
" start_idx, end_idx = int(start_t * sr), int(end_t * sr)\n",
" segment = audio[start_idx:end_idx]\n",
" save_audio_file(segment, save_path, sr=sr)\n",
" return segment\n",
"\n",
"def get_segment_fname(id: str, segment_index: int):\n",
" return f\"{id}_{segment_index}\"\n",
"\n",
"def make_dataset():\n",
" data_list = []\n",
" for id in tqdm(processed_songs):\n",
" # Get vocals audio\n",
" audio_path = os.path.join(path_vocals, f\"{id}.wav\")\n",
" audio, sr = librosa.load(audio_path, sr=TARGET_SR)\n",
" # Get annotations\n",
" entry = dali_data[id]\n",
" anno = entry.annotations['annot']\n",
" lines = anno['lines']\n",
" # Process each line\n",
" for seg_idx, line in enumerate(lines):\n",
" # Extract segment\n",
" segment_time = line['time']\n",
" extract_audio_segment(\n",
" audio,\n",
" segment_time,\n",
" save_path=os.path.join(path_segments, get_segment_fname(id, seg_idx))\n",
" )\n",
" # Add annotation to dataset\n",
" data_list.append((id, seg_idx, line['text']))\n",
" \n",
" df = pd.DataFrame(data_list, columns=['id', 'segment_index', 'text'])\n",
" return df\n",
"\n",
"df = make_dataset()\n",
"df.to_csv(csv_path, index=False)\n",
"df.head(3)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Make training dataset & data cleaning"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from num2words import num2words\n",
"import re\n",
"import numpy as np\n",
"\n",
"allowed_chars = set('abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ \\'')\n",
"\n",
"def clean_lyrics(lyrics: str) -> str:\n",
" # Find all numbers in the text using regular expression\n",
" num_regex = r'\\d+'\n",
" numbers = re.findall(num_regex, lyrics)\n",
" \n",
" # Convert each number to its word form\n",
" for num in numbers:\n",
" word_form = num2words(int(num))\n",
" lyrics = lyrics.replace(num, word_form)\n",
"\n",
" cleaned = \"\"\n",
" for char in lyrics:\n",
" if char in allowed_chars:\n",
" cleaned += char\n",
" elif char == '-':\n",
" cleaned += ' '\n",
" return cleaned\n",
"\n",
"dataset_csv_path = \"dataset.csv\"\n",
"def transform_dataset(csv_path, path_segments, lang='english'):\n",
" df = pd.read_csv(csv_path)\n",
" df['path'] = df.agg(lambda x: f\"{path_segments}/{x['id']}_{str(x['segment_index'])}.wav\", axis=1)\n",
" delete_idx = []\n",
" for idx, row in tqdm(df.iterrows()):\n",
" song_id = row['id']\n",
" # Get annotations\n",
" entry = dali_data[song_id]\n",
" meta = entry.info['metadata']\n",
" song_lang = meta['language']\n",
" # Skip different lang\n",
" if song_lang != lang:\n",
" delete_idx.append(idx)\n",
" continue\n",
" # Clean invalid data\n",
" if not isinstance(row['text'], str) or len(row['text']) == 0:\n",
" delete_idx.append(idx)\n",
" continue\n",
" # Check for audio validity\n",
" y, sr = librosa.load(row['path'], sr=TARGET_SR)\n",
" yt, _ = librosa.effects.trim(y, top_db=30)\n",
" duration = librosa.get_duration(y=yt, sr=sr)\n",
" if duration > 3.1 or duration < 0.15: # CUDA out of memory\n",
" delete_idx.append(idx)\n",
" continue \n",
"\n",
" df = df.drop(delete_idx)\n",
" df = df.drop(['id', 'segment_index'], axis=1)\n",
"\n",
" # Text cleaning\n",
" df['text'] = df['text'].map(clean_lyrics)\n",
" df.replace('', np.nan, inplace=True)\n",
" df.dropna(inplace=True)\n",
" return df\n",
"\n",
"df = transform_dataset(csv_path, path_segments)\n",
"df.to_csv(dataset_csv_path, index=False)\n",
"df.head(10)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Finetune"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Load dataset"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from datasets import load_dataset, Audio, Dataset\n",
"\n",
"dataset_csv_path = \"dataset.csv\"\n",
"dataset = Dataset.from_csv(dataset_csv_path)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Sample of dataset loaded"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import IPython.display as ipd\n",
"import numpy as np\n",
"import random\n",
"import librosa\n",
"\n",
"rand_int = random.randint(0, len(dataset))\n",
"\n",
"def audio_processing(fp):\n",
" y, sr = librosa.load(fp, sr=TARGET_SR)\n",
" return y\n",
"\n",
"print(dataset[rand_int][\"text\"])\n",
"ipd.Audio(data=audio_processing(dataset[rand_int][\"path\"]), autoplay=True, rate=16000)\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Data cleaning"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import re\n",
"chars_to_ignore_regex = '[\\,\\?\\.\\!\\-\\;\\:\\\"]'\n",
"\n",
"\n",
"def remove_special_characters(batch):\n",
" batch[\"text\"] = re.sub(chars_to_ignore_regex, '', batch[\"text\"]).lower()\n",
" return batch\n",
"\n",
"dataset = dataset.map(remove_special_characters)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import json\n",
"\n",
"def make_vocab(dataset):\n",
" vocab = set()\n",
" for sample in dataset['text']:\n",
" for ch in sample:\n",
" vocab.add(ch)\n",
" vocab_dict = {v: k for k, v in enumerate(sorted(vocab))}\n",
" vocab_dict[\"|\"] = vocab_dict[\" \"]\n",
" del vocab_dict[\" \"]\n",
" vocab_dict[\"[UNK]\"] = len(vocab_dict)\n",
" vocab_dict[\"[PAD]\"] = len(vocab_dict)\n",
" with open('vocab.json', 'w') as vocab_file:\n",
" json.dump(vocab_dict, vocab_file)\n",
" return vocab_dict\n",
"\n",
"vocab_dict = make_vocab(dataset)\n",
"with open('vocab.json', 'r') as f:\n",
" vocab_dict = json.load(f)\n",
"\n",
"\n",
"print(vocab_dict)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "lsync",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.12"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}
+248
View File
@@ -0,0 +1,248 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Demo"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Using pre-trained model (facebook/wav2vec2-large-960h-lv60-self)"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"audio_path = \"dataset/hensen/umbrella/umbrella.wav\"\n",
"lyrics_path = \"dataset/hensen/umbrella/umbrella.txt\""
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Some weights of Wav2Vec2ForCTC were not initialized from the model checkpoint at facebook/wav2vec2-large-960h-lv60-self and are newly initialized: ['wav2vec2.masked_spec_embed']\n",
"You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"[00:00.00] <00:27.53> you <00:27.90> have <00:28.21> my <00:28.42> heart\n",
"[00:29.40] <00:29.46> and <00:29.62> we'll <00:29.98> never <00:30.32> be <00:30.62> worlds <00:31.12> apart\n",
"[00:32.32] <00:32.35> maybe <00:33.07> in <00:33.38> magazines\n",
"[00:35.11] <00:35.16> but <00:35.48> you'll <00:35.75> still <00:36.14> be <00:36.39> my <00:36.61> star\n",
"[00:37.84] <00:37.88> baby <00:38.38> 'cause <00:38.98> in <00:39.20> the <00:39.42> dark\n",
"[00:40.57] <00:40.61> you <00:40.92> can't <00:41.30> see <00:41.61> shiny <00:42.14> cars\n",
"[00:43.14] <00:43.17> and <00:43.35> that's <00:43.70> when <00:44.03> you <00:44.38> need <00:44.74> me <00:44.96> there\n",
"[00:46.06] <00:46.10> with <00:46.46> you <00:46.88> i'll <00:47.16> always <00:47.82> share\n",
"[00:49.14] <00:49.17> because <00:50.56> when <00:50.74> the <00:50.92> sun <00:51.21> shines <00:51.64> we <00:51.93> shine <00:52.32> together\n",
"[00:53.14] <00:53.38> told <00:53.74> you <00:54.10> i'll <00:54.42> be <00:54.78> here <00:55.06> forever\n",
"[00:56.03] <00:56.10> said <00:56.56> i'll <00:56.92> always <00:57.50> be <00:57.85> your <00:57.98> friend\n",
"[00:58.50] <00:58.53> took <00:58.75> an <00:58.96> oath <00:59.20> gonna <00:59.46> stick <00:59.78> it <01:00.02> out <01:00.21> till <01:00.57> the <01:00.84> end\n",
"[01:01.56] <01:01.62> now <01:01.92> that <01:02.10> it's <01:02.34> rainin' <01:03.00> more <01:03.30> than <01:03.57> ever\n",
"[01:04.32] <01:04.34> know <01:04.71> that <01:05.07> we'll <01:05.37> still <01:05.78> have <01:06.14> each <01:06.34> other\n",
"[01:07.09] <01:07.14> you <01:07.29> can <01:07.46> stand <01:07.85> under <01:08.18> my <01:08.62> umbrella\n",
"[01:09.84] <01:09.87> you <01:10.01> can <01:10.18> stand <01:10.60> under <01:10.92> my <01:11.35> umbrella\n",
"[01:12.62] <01:12.71> ella <01:13.37> ella <01:14.15> ae <01:14.76> ae <01:14.98> ae\n",
"[01:16.00] <01:16.06> under <01:16.43> my <01:16.90> umbrella <01:18.20> ella <01:18.87> ella <01:19.64> ae <01:20.32> ae <01:21.00> ae\n",
"[01:21.54] <01:21.60> under <01:21.93> my <01:22.40> umbrella <01:23.70> ella <01:24.37> ella <01:25.15> ae <01:25.79> ae <01:26.51> ae\n",
"[01:27.04] <01:27.10> under <01:27.48> my <01:27.92> umbrella <01:29.21> ella <01:29.92> ella <01:30.65> ae <01:31.32> ae <01:32.01> ae <01:32.07> ae <01:32.68> ae <01:33.35> ae\n",
"[01:33.56] <01:33.60> these <01:34.00> fancy <01:34.50> things\n",
"[01:35.65] <01:35.70> will <01:36.06> never <01:36.37> come <01:36.84> in <01:37.06> between\n",
"[01:38.37] <01:38.42> you're <01:38.76> part <01:39.01> of <01:39.18> my <01:39.57> entity\n",
"[01:41.14] <01:41.18> here <01:41.54> for <01:42.00> infinity\n",
"[01:43.90] <01:43.93> when <01:44.12> the <01:44.32> war <01:44.68> has <01:44.98> took <01:45.32> it's <01:45.56> part\n",
"[01:46.65] <01:46.68> when <01:46.85> the <01:47.06> world <01:47.37> has <01:47.74> dealt <01:48.07> it's <01:48.34> cards\n",
"[01:49.78] <01:49.90> if <01:50.15> the <01:50.48> hand <01:50.87> is <01:51.01> hard\n",
"[01:52.35] <01:52.37> together <01:52.90> we'll <01:53.21> mend <01:53.57> your <01:53.79> heart\n",
"[01:55.24] <01:55.28> because <01:56.68> when <01:56.85> the <01:57.04> sun <01:57.35> shines <01:57.76> we <01:58.06> shine <01:58.43> together\n",
"[01:59.24] <01:59.40> told <01:59.87> you <02:00.26> i'll <02:00.51> be <02:00.84> here <02:01.15> forever\n",
"[02:02.15] <02:02.20> said <02:02.68> i'll <02:03.01> always <02:03.62> be <02:03.96> your <02:04.07> friend\n",
"[02:04.64] <02:04.65> took <02:04.84> an <02:05.07> oath <02:05.29> gonna <02:05.56> stick <02:05.90> it <02:06.12> out <02:06.34> till <02:06.70> the <02:06.93> end\n",
"[02:07.68] <02:07.70> now <02:08.06> that <02:08.19> it's <02:08.46> rainin' <02:09.12> more <02:09.42> than <02:09.69> ever\n",
"[02:10.46] <02:10.47> know <02:10.84> that <02:11.19> we'll <02:11.52> still <02:11.87> have <02:12.25> each <02:12.46> other\n",
"[02:13.21] <02:13.25> you <02:13.42> can <02:13.59> stand <02:13.97> under <02:14.30> my <02:14.84> umbrella\n",
"[02:15.93] <02:15.97> you <02:16.14> can <02:16.28> stand <02:16.70> under <02:17.03> my <02:17.50> umbrella\n",
"[02:18.74] <02:18.80> ella <02:19.47> ella <02:20.25> ae <02:20.92> ae <02:21.62> ae\n",
"[02:22.12] <02:22.18> under <02:22.56> my <02:23.02> umbrella <02:24.30> ella <02:25.00> ella <02:25.75> ae <02:26.46> ae <02:27.15> ae\n",
"[02:27.68] <02:27.74> under <02:28.08> my <02:28.52> umbrella <02:29.86> ella <02:30.47> ella <02:31.18> ae <02:31.90> ae <02:32.62> ae\n",
"[02:33.14] <02:33.20> under <02:33.58> my <02:34.02> umbrella <02:35.34> ella <02:36.00> ella <02:36.78> ae <02:37.43> ae <02:38.15> ae\n",
"[02:38.65] <02:38.78> ae <02:39.47> ae <02:39.86> ae\n",
"[02:40.12] <02:40.15> you <02:40.42> can <02:40.70> run <02:41.12> into <02:41.71> my <02:42.25> arms\n",
"[02:42.86] <02:42.96> it's <02:43.25> okay <02:43.78> don't <02:44.12> be <02:44.53> alarmed <02:45.31> come <02:45.78> into <02:46.70> me\n",
"[02:47.14] <02:47.15> there's <02:47.50> no <02:47.75> distance <02:48.59> in <02:49.12> between <02:50.59> our <02:50.93> love\n",
"[02:51.47] <02:51.50> so <02:51.81> go <02:52.08> on <02:52.24> and <02:52.50> let <02:52.84> the <02:53.20> rain <02:53.87> pour\n",
"[02:54.64] <02:54.71> i'll <02:54.96> be <02:55.42> all <02:55.65> you <02:55.97> need <02:56.37> and <02:56.70> more\n",
"[02:59.74] <03:01.43> because <03:02.80> when <03:02.96> the <03:03.15> sun <03:03.46> shines <03:03.87> we <03:04.18> shine <03:04.53> together\n",
"[03:05.56] <03:05.59> told <03:05.97> you <03:06.37> i'll <03:06.64> be <03:07.03> here <03:07.28> forever\n",
"[03:08.30] <03:08.34> said <03:08.78> i'll <03:09.18> always <03:09.75> be <03:10.21> your <03:10.34> friend <03:10.75> took <03:10.97> an <03:11.20> oath <03:11.43> gonna <03:11.78> stick <03:12.03> it <03:12.25> out <03:12.48> till <03:12.84> the <03:13.09> end\n",
"[03:13.81] <03:13.86> now <03:14.18> that <03:14.37> it's <03:14.75> rainin' <03:15.24> more <03:15.52> than <03:15.80> ever\n",
"[03:16.58] <03:16.62> know <03:16.96> that <03:17.31> we'll <03:17.62> still <03:18.00> have <03:18.36> each <03:18.58> other\n",
"[03:19.34] <03:19.36> you <03:19.53> can <03:19.70> stand <03:20.09> under <03:20.42> my <03:20.84> umbrella\n",
"[03:22.08] <03:22.09> you <03:22.25> can <03:22.42> stand <03:22.84> under <03:23.15> my <03:23.58> umbrella\n",
"[03:24.84] <03:24.92> ella <03:25.59> ella <03:26.37> ae <03:27.06> ae <03:27.74> ae\n",
"[03:28.25] <03:28.31> under <03:28.68> my <03:29.12> umbrella <03:30.42> ella <03:31.12> ella <03:31.86> ae <03:32.53> ae <03:33.24> ae\n",
"[03:33.75] <03:33.84> under <03:34.18> my <03:34.64> umbrella <03:35.92> ella <03:36.62> ella <03:37.37> ae <03:38.02> ae <03:38.75> ae\n",
"[03:39.25] <03:39.31> under <03:39.70> my <03:40.14> umbrella <03:41.46> ella <03:42.12> ella <03:42.59> ae <03:42.90> ae <03:43.56> ae <03:44.25> ae <03:44.90> ae <03:45.59> ae\n",
"[03:45.81] <03:45.98> it's <03:46.24> rainin' <03:46.90> raining\n",
"[03:47.86] <03:48.00> oh <03:48.28> baby <03:48.70> it's <03:48.96> rainin' <03:49.64> raining\n",
"[03:50.65] <03:50.70> baby <03:51.40> come <03:51.78> here <03:52.50> to <03:52.75> me\n",
"[03:53.25] <03:53.56> come <03:53.70> here <03:54.12> to <03:54.40> me\n",
"[03:54.48] <03:54.64> it's <03:55.58> raining <03:56.09> raining\n",
"[03:56.40] <03:56.46> oh <03:56.86> baby <03:56.96> it's <03:57.24> raining <03:57.65> raining\n",
"[03:57.80] <03:57.84> you <03:57.92> can <03:58.36> always <03:58.93> come <03:59.34> here\n"
]
}
],
"source": [
"from lsync import LyricsSync\n",
"\n",
"lsync = LyricsSync()\n",
"words, lrc = lsync.sync(audio_path, lyrics_path)\n",
"print(lrc)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Using Fine-tuned model\n",
"\n",
"Please update your model's id and path in lsync/config.py"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"YOUR_MODEL_ID = \"en-finetuned-base\"\n",
"YOUR_MODEL_BLANK_ID = 29"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[00:00.00] <00:27.53> you <00:27.85> have <00:28.17> my <00:28.35> heart\n",
"[00:29.17] <00:29.46> and <00:29.60> we'll <00:29.96> never <00:30.28> be <00:30.58> worlds <00:30.98> apart\n",
"[00:31.87] <00:32.35> maybe <00:33.03> in <00:33.35> magazines\n",
"[00:35.07] <00:35.11> but <00:35.46> you'll <00:35.74> still <00:36.11> be <00:36.42> my <00:36.56> star\n",
"[00:37.38] <00:37.88> baby <00:38.50> 'cause <00:38.93> in <00:39.20> the <00:39.39> dark\n",
"[00:40.14] <00:40.61> you <00:40.92> can't <00:41.28> see <00:41.57> shiny <00:42.11> cars\n",
"[00:42.96] <00:43.20> and <00:43.35> that's <00:43.67> when <00:44.02> you <00:44.34> need <00:44.70> me <00:44.88> there\n",
"[00:45.60] <00:46.10> with <00:46.39> you <00:46.80> i'll <00:47.11> always <00:47.78> share\n",
"[00:48.56] <00:49.17> because <00:50.56> when <00:50.71> the <00:50.92> sun <00:51.21> shines <00:51.62> we <00:51.93> shine <00:52.30> together\n",
"[00:53.02] <00:53.34> told <00:53.70> you <00:54.06> i'll <00:54.39> be <00:54.71> here <00:55.03> forever\n",
"[00:55.80] <00:56.10> said <00:56.52> i'll <00:56.84> always <00:57.48> be <00:57.80> your <00:57.98> friend\n",
"[00:58.50] <00:58.52> took <00:58.74> an <00:58.92> oath <00:59.16> gonna <00:59.43> stick <00:59.74> it <00:59.89> out <01:00.20> till <01:00.53> the <01:00.75> end\n",
"[01:01.28] <01:01.60> now <01:01.92> that <01:02.10> it's <01:02.32> rainin' <01:02.96> more <01:03.30> than <01:03.52> ever\n",
"[01:04.06] <01:04.37> know <01:04.70> that <01:05.04> we'll <01:05.35> still <01:05.71> have <01:06.07> each <01:06.28> other\n",
"[01:06.82] <01:07.14> you <01:07.29> can <01:07.46> stand <01:07.84> under <01:08.15> my <01:08.56> umbrella\n",
"[01:09.54] <01:09.87> you <01:10.01> can <01:10.20> stand <01:10.57> under <01:10.90> my <01:11.29> umbrella\n",
"[01:12.42] <01:12.65> ella <01:13.35> ella <01:14.04> ae <01:14.70> ae <01:15.40> ae\n",
"[01:15.82] <01:16.04> under <01:16.40> my <01:16.78> umbrella <01:18.15> ella <01:18.85> ella <01:19.46> ae <01:20.20> ae <01:20.92> ae\n",
"[01:21.32] <01:21.57> under <01:21.92> my <01:22.32> umbrella <01:23.68> ella <01:24.34> ella <01:24.98> ae <01:25.70> ae <01:26.32> ae\n",
"[01:26.79] <01:27.07> under <01:27.43> my <01:27.82> umbrella <01:28.28> ella <01:29.18> ella <01:29.90> ae <01:30.06> ae <01:30.56> ae <01:31.21> ae <01:32.57> ae <01:33.29> ae\n",
"[01:33.54] <01:33.57> these <01:33.98> fancy <01:34.48> things\n",
"[01:35.21] <01:35.71> will <01:36.04> never <01:36.35> come <01:36.76> in <01:37.04> between\n",
"[01:37.92] <01:38.42> you're <01:38.76> part <01:39.00> of <01:39.15> my <01:39.50> entity\n",
"[01:40.71] <01:41.18> here <01:41.51> for <01:42.00> infinity\n",
"[01:43.37] <01:43.93> when <01:44.12> the <01:44.29> war <01:44.64> has <01:44.98> took <01:45.28> it's <01:45.51> part\n",
"[01:46.40] <01:46.70> when <01:46.85> the <01:47.01> world <01:47.35> has <01:47.71> dealt <01:48.06> it's <01:48.28> cards\n",
"[01:49.35] <01:49.84> if <01:50.14> the <01:50.43> hand <01:50.84> is <01:50.98> hard\n",
"[01:51.84] <01:52.37> together <01:52.90> we'll <01:53.21> mend <01:53.56> your <01:53.79> heart\n",
"[01:54.96] <01:55.28> because <01:56.68> when <01:56.84> the <01:57.04> sun <01:57.34> shines <01:57.74> we <01:58.04> shine <01:58.42> together\n",
"[01:59.46] <01:59.48> told <01:59.84> you <02:00.15> i'll <02:00.50> be <02:00.82> here <02:01.14> forever\n",
"[02:01.90] <02:02.21> said <02:02.62> i'll <02:02.93> always <02:03.57> be <02:03.92> your <02:04.07> friend\n",
"[02:04.60] <02:04.64> took <02:04.84> an <02:05.01> oath <02:05.28> gonna <02:05.62> stick <02:05.85> it <02:06.04> out <02:06.34> till <02:06.68> the <02:06.88> end\n",
"[02:07.43] <02:07.71> now <02:08.06> that <02:08.24> it's <02:08.43> rainin' <02:09.08> more <02:09.42> than <02:09.64> ever\n",
"[02:10.18] <02:10.50> know <02:10.81> that <02:11.18> we'll <02:11.50> still <02:11.84> have <02:12.21> each <02:12.40> other\n",
"[02:12.96] <02:13.25> you <02:13.42> can <02:13.58> stand <02:13.97> under <02:14.28> my <02:14.65> umbrella\n",
"[02:15.68] <02:16.00> you <02:16.12> can <02:16.24> stand <02:16.68> under <02:17.02> my <02:17.40> umbrella\n",
"[02:18.53] <02:18.78> ella <02:19.46> ella <02:20.14> ae <02:20.81> ae <02:21.43> ae\n",
"[02:21.93] <02:22.15> under <02:22.53> my <02:22.92> umbrella <02:24.28> ella <02:24.97> ella <02:25.65> ae <02:26.34> ae <02:27.06> ae\n",
"[02:27.43] <02:27.71> under <02:28.03> my <02:28.43> umbrella <02:29.84> ella <02:30.46> ella <02:31.15> ae <02:31.80> ae <02:32.52> ae\n",
"[02:32.90] <02:33.18> under <02:33.53> my <02:33.92> umbrella <02:35.30> ella <02:35.96> ella <02:36.68> ae <02:37.34> ae <02:38.06> ae\n",
"[02:38.62] <02:38.68> ae <02:39.42> ae <02:39.75> ae\n",
"[02:40.09] <02:40.15> you <02:40.42> can <02:40.68> run <02:41.08> into <02:41.68> my <02:42.08> arms\n",
"[02:42.84] <02:42.92> it's <02:43.21> okay <02:43.75> don't <02:44.08> be <02:44.43> alarmed <02:45.30> come <02:45.70> into <02:46.65> me\n",
"[02:47.40] <02:47.42> there's <02:47.64> no <02:47.70> distance <02:48.50> in <02:49.08> between <02:50.56> our <02:50.90> love\n",
"[02:51.43] <02:51.47> so <02:51.81> go <02:52.08> on <02:52.20> and <02:52.47> let <02:52.81> the <02:53.18> rain <02:53.81> pour\n",
"[02:54.53] <02:54.64> i'll <02:54.96> be <02:55.31> all <02:55.64> you <02:55.93> need <02:56.34> and <02:56.64> more\n",
"[03:01.37] <03:01.42> because <03:02.80> when <03:02.96> the <03:03.14> sun <03:03.46> shines <03:03.86> we <03:04.15> shine <03:04.52> together\n",
"[03:05.56] <03:05.58> told <03:05.93> you <03:06.28> i'll <03:06.62> be <03:06.96> here <03:07.28> forever\n",
"[03:08.03] <03:08.34> said <03:08.74> i'll <03:09.08> always <03:09.70> be <03:10.03> your <03:10.20> friend <03:10.75> took <03:10.96> an <03:11.15> oath <03:11.40> gonna <03:11.68> stick <03:11.98> it <03:12.18> out <03:12.46> till <03:12.80> the <03:13.02> end\n",
"[03:13.53] <03:13.84> now <03:14.18> that <03:14.36> it's <03:14.53> rainin' <03:15.18> more <03:15.52> than <03:15.75> ever\n",
"[03:16.30] <03:16.62> know <03:16.93> that <03:17.28> we'll <03:17.59> still <03:17.96> have <03:18.31> each <03:18.50> other\n",
"[03:19.06] <03:19.37> you <03:19.52> can <03:19.70> stand <03:20.08> under <03:20.37> my <03:20.78> umbrella\n",
"[03:21.78] <03:22.09> you <03:22.25> can <03:22.42> stand <03:22.80> under <03:23.14> my <03:23.52> umbrella\n",
"[03:24.65] <03:24.90> ella <03:25.59> ella <03:26.20> ae <03:26.92> ae <03:27.62> ae\n",
"[03:28.21] <03:28.28> under <03:28.65> my <03:29.02> umbrella <03:30.40> ella <03:31.08> ella <03:31.75> ae <03:32.42> ae <03:33.14> ae\n",
"[03:33.53] <03:33.81> under <03:34.14> my <03:34.53> umbrella <03:35.90> ella <03:36.58> ella <03:37.28> ae <03:37.92> ae <03:38.64> ae\n",
"[03:39.02] <03:39.30> under <03:39.65> my <03:40.06> umbrella <03:41.42> ella <03:42.08> ella <03:42.80> ae <03:43.46> ae <03:44.14> ae <03:44.84> ae <03:45.21> ae <03:45.53> ae\n",
"[03:45.80] <03:45.90> it's <03:46.24> rainin' <03:46.92> raining\n",
"[03:47.46] <03:47.93> oh <03:48.28> baby <03:48.65> it's <03:48.96> rainin' <03:49.64> raining\n",
"[03:50.24] <03:50.70> baby <03:51.36> come <03:51.75> here <03:52.43> to <03:52.75> me\n",
"[03:52.92] <03:53.31> come <03:53.93> here <03:54.12> to <03:54.40> me\n",
"[03:54.48] <03:54.52> it's <03:54.93> raining <03:55.53> raining\n",
"[03:56.52] <03:56.58> oh <03:56.90> baby <03:57.02> it's <03:57.21> raining <03:57.92> raining\n",
"[03:58.53] <03:58.59> you <03:58.90> can <03:59.30> always <03:59.46> come <03:59.59> here\n"
]
}
],
"source": [
"from lsync import LyricsSync\n",
"\n",
"lsync = LyricsSync(lang=YOUR_MODEL_ID, blank_id=YOUR_MODEL_BLANK_ID)\n",
"words, lrc = lsync.sync(audio_path, lyrics_path)\n",
"print(lrc)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "lsync",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.12"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}
+259
View File
@@ -0,0 +1,259 @@
name: lsync
channels:
- pytorch
- nvidia
- defaults
- conda-forge
dependencies:
- _libgcc_mutex=0.1=conda_forge
- _openmp_mutex=4.5=2_kmp_llvm
- autopep8=1.6.0=pyhd3eb1b0_1
- backcall=0.2.0=pyhd3eb1b0_0
- blas=1.0=mkl
- bzip2=1.0.8=h7f98852_4
- ca-certificates=2022.12.7=ha878542_0
- colorama=0.4.6=pyhd8ed1ab_0
- cuda-cudart=11.7.99=0
- cuda-cupti=11.7.101=0
- cuda-libraries=11.7.1=0
- cuda-nvrtc=11.7.99=0
- cuda-nvtx=11.7.91=0
- cuda-runtime=11.7.1=0
- decorator=5.1.1=pyhd3eb1b0_0
- entrypoints=0.4=py37h06a4308_0
- ffmpeg=4.3.2=h37c90e5_3
- freetype=2.12.1=hca18f0e_1
- gmp=6.2.1=h58526e2_0
- gnutls=3.6.13=h79a8f9a_0
- icu=72.1=hcb278e6_0
- ipython_genutils=0.2.0=pyhd3eb1b0_1
- ipywidgets=8.0.6=pyhd8ed1ab_0
- jupyter_client=5.3.4=py37_0
- jupyter_core=4.5.0=py_0
- jupyterlab_widgets=3.0.7=pyhd8ed1ab_0
- lame=3.100=h166bdaf_1003
- ld_impl_linux-64=2.40=h41732ed_0
- libcublas=11.10.3.66=0
- libcufft=10.7.2.124=h4fbf590_0
- libcufile=1.6.0.25=0
- libcurand=10.3.2.56=0
- libcusolver=11.4.0.1=0
- libcusparse=11.7.4.91=0
- libffi=3.4.2=h7f98852_5
- libgcc-ng=12.2.0=h65d4601_19
- libhwloc=2.9.1=hd6dc26d_0
- libiconv=1.17=h166bdaf_0
- libnpp=11.7.4.75=0
- libnsl=2.0.0=h7f98852_0
- libnvjpeg=11.8.0.2=0
- libopus=1.3.1=h7f98852_1
- libpng=1.6.39=h753d276_0
- libsodium=1.0.18=h7b6447c_0
- libsqlite=3.40.0=h753d276_0
- libstdcxx-ng=12.2.0=h46fd767_19
- libvpx=1.7.0=h439df22_0
- libxml2=2.10.4=hfdac1af_0
- libzlib=1.2.13=h166bdaf_4
- llvm-openmp=16.0.1=h417c0b6_0
- matplotlib-inline=0.1.6=py37h06a4308_0
- mkl=2021.4.0=h8d4b97c_729
- mkl-service=2.4.0=py37h402132d_0
- mkl_fft=1.3.1=py37h3e078e5_1
- mkl_random=1.2.2=py37h219a48f_0
- ncurses=6.3=h27087fc_1
- nettle=3.4.1=h1bed415_1002
- numpy=1.21.5=py37h6c91a56_3
- numpy-base=1.21.5=py37ha15fc14_3
- openh264=2.1.1=h780b84a_0
- openssl=3.1.0=h0b41bf4_0
- parso=0.8.3=pyhd3eb1b0_0
- pexpect=4.8.0=pyhd3eb1b0_3
- pickleshare=0.7.5=pyhd3eb1b0_1003
- pip=23.0.1=pyhd8ed1ab_0
- ptyprocess=0.7.0=pyhd3eb1b0_2
- pycodestyle=2.10.0=py37h06a4308_0
- python=3.7.12=hf930737_100_cpython
- python-dateutil=2.8.2=pyhd3eb1b0_0
- python_abi=3.7=3_cp37m
- pytorch=1.13.1=py3.7_cuda11.7_cudnn8.5.0_0
- pytorch-cuda=11.7=h778d358_3
- pytorch-mutex=1.0=cuda
- readline=8.2=h8228510_1
- six=1.16.0=pyh6c4a22f_0
- sqlite=3.40.0=h4ff8645_0
- tbb=2021.9.0=hf52228f_0
- tk=8.6.12=h27826a3_0
- toml=0.10.2=pyhd3eb1b0_0
- torchaudio=0.13.1=py37_cu117
- tornado=6.2=py37h5eee18b_0
- tqdm=4.65.0=pyhd8ed1ab_1
- typing_extensions=4.5.0=pyha770c72_0
- wheel=0.40.0=pyhd8ed1ab_0
- widgetsnbextension=4.0.7=pyhd8ed1ab_0
- x264=1!161.3030=h7f98852_1
- xz=5.2.6=h166bdaf_0
- zeromq=4.3.4=h2531618_0
- zlib=1.2.13=h166bdaf_4
- zstd=1.5.2=h3eb15da_6
- pip:
- accelerate==0.19.0
- aiohttp==3.8.4
- aiosignal==1.3.1
- antlr4-python3-runtime==4.9.3
- anyio==3.6.2
- appdirs==1.4.4
- argon2-cffi==21.3.0
- argon2-cffi-bindings==21.2.0
- async-timeout==4.0.2
- asynctest==0.13.0
- attrs==22.2.0
- audioread==3.0.0
- beautifulsoup4==4.12.2
- bleach==6.0.0
- certifi==2022.12.7
- cffi==1.15.1
- charset-normalizer==3.1.0
- click==8.1.3
- cloudpickle==2.2.1
- colorlog==6.7.0
- cycler==0.11.0
- cython==0.29.34
- dali-dataset==1.1
- datasets==2.12.0
- debugpy==1.6.7
- defusedxml==0.7.1
- demucs==4.0.0
- diffq==0.2.3
- dill==0.3.6
- docker-pycreds==0.4.0
- docopt==0.6.2
- dora-search==0.1.11
- einops==0.6.0
- evaluate==0.4.0
- exceptiongroup==1.1.1
- fastjsonschema==2.16.3
- ffmpeg-python==0.2.0
- filelock==3.11.0
- fonttools==4.38.0
- frozenlist==1.3.3
- fsspec==2023.1.0
- future==0.18.3
- gitdb==4.0.10
- gitpython==3.1.31
- huggingface-hub==0.14.1
- hydra-colorlog==1.2.0
- hydra-core==1.3.2
- hyperpyyaml==1.2.0
- hypothesis==6.71.0
- idna==3.4
- importlib-metadata==4.13.0
- importlib-resources==5.12.0
- inquirerpy==0.3.4
- ipykernel==6.16.2
- ipython==7.34.0
- jedi==0.18.2
- jieba==0.42.1
- jinja2==3.1.2
- jiwer==3.0.1
- joblib==1.2.0
- jsonschema==4.17.3
- julius==0.2.7
- jupyter==1.0.0
- jupyter-client==7.4.9
- jupyter-console==6.6.3
- jupyter-core==4.12.0
- jupyter-server==1.24.0
- jupyterlab-pygments==0.2.2
- kiwisolver==1.4.4
- lameenc==1.4.2
- lazy-loader==0.2
- librosa==0.10.0.post2
- llvmlite==0.39.1
- markupsafe==2.1.2
- matplotlib==3.5.3
- mistune==2.0.5
- msgpack==1.0.5
- multidict==6.0.4
- multiprocess==0.70.14
- musdb==0.4.0
- museval==0.4.0
- nbclassic==0.5.5
- nbclient==0.7.3
- nbconvert==7.3.1
- nbformat==5.8.0
- nest-asyncio==1.5.6
- notebook==6.5.4
- notebook-shim==0.2.2
- num2words==0.5.12
- numba==0.56.4
- omegaconf==2.3.0
- openunmix==1.2.1
- packaging==23.1
- pandas==1.3.5
- pandocfilters==1.5.0
- pathtools==0.1.2
- pfzy==0.3.4
- pillow==9.5.0
- pipreqs==0.4.13
- pkgutil-resolve-name==1.3.10
- pooch==1.6.0
- prometheus-client==0.16.0
- prompt-toolkit==3.0.38
- protobuf==4.22.3
- psutil==5.9.4
- pyaml==21.10.1
- pyarrow==11.0.0
- pycparser==2.21
- pyctcdecode==0.5.0
- pydub==0.25.1
- pygments==2.15.0
- pygtrie==2.5.0
- pyparsing==3.0.9
- pyrsistent==0.19.3
- pytz==2023.3
- pyyaml==6.0
- pyzmq==25.0.2
- qtconsole==5.4.2
- qtpy==2.3.1
- rapidfuzz==2.13.7
- regex==2022.10.31
- requests==2.28.2
- responses==0.18.0
- retrying==1.3.4
- ruamel-yaml==0.17.24
- ruamel-yaml-clib==0.2.7
- scikit-learn==1.0.2
- scipy==1.7.3
- seaborn==0.12.2
- send2trash==1.8.0
- sentencepiece==0.1.99
- sentry-sdk==1.19.1
- setproctitle==1.3.2
- setuptools==67.6.1
- simplejson==3.19.1
- smmap==5.0.0
- sniffio==1.3.0
- sortedcontainers==2.4.0
- soundfile==0.12.1
- soupsieve==2.4
- soxr==0.3.5
- speechbrain==0.5.14
- stempeg==0.2.3
- submitit==1.4.5
- terminado==0.17.1
- threadpoolctl==3.1.0
- tinycss2==1.2.1
- tokenizers==0.13.3
- traitlets==5.9.0
- transformers==4.29.1
- treetable==0.2.5
- urllib3==1.26.15
- wandb==0.14.2
- wcwidth==0.2.6
- webencodings==0.5.1
- websocket-client==1.5.1
- xxhash==3.2.0
- yarg==0.1.9
- yarl==1.8.2
- youtube-dl==2021.12.17
- zipp==3.15.0
+12
View File
@@ -0,0 +1,12 @@
import DALI as dali_code
import os
dali_data_path = os.path.abspath('dataset/DALI/v1')
path_audio = os.path.abspath('dataset/DALI/audio')
dali_data = dali_code.get_the_DALI_dataset(dali_data_path, skip=[], keep=[])
dali_info = dali_code.get_info(dali_data_path + '/info/DALI_DATA_INFO.gz')
# Get Audio
downloaded_audio = [os.path.splitext(x)[0] for x in os.listdir(path_audio)]
errors = dali_code.get_audio(dali_info, path_audio, skip=downloaded_audio, keep=[])
+46
View File
@@ -0,0 +1,46 @@
from .alignment import Aligner
from .lyrics_processor import LyricsProcessor
from .util import read_text, save_lrc, save_words, save_audio
from .config import TARGET_SR
from .lrc_formatter import LrcFormatter
import os
class LyricsSync():
def __init__(self, lang="en-US", blank_id=0) -> None:
# Deferred so that merely importing this module doesn't pull in
# transformers/demucs at module load time (those fail loudly if the
# installed transformers version is broken or incompatible).
from .voice_extractor import VoiceExtractor
from .phoneme_recognizer import PhonemeRecognizer
self.ve = VoiceExtractor()
self.phone_rec = PhonemeRecognizer(lang=lang)
self.lp = LyricsProcessor(lang=lang)
self.blank_id = blank_id
def sync(self, audio_fn: str, text_fn: str, save=True, timing_precision: str = 'word'):
audio_name = os.path.splitext(os.path.basename(audio_fn))[0]
vocals = self.ve.extract_voice(audio_fn)
if save:
save_audio(vocals, audio_name, TARGET_SR)
emission, _, _, frame_duration = self.phone_rec.recognize(vocals)
# Preprocessing & tokenization
tokens = self.phone_rec.tokenize(text_fn)
path = Aligner.align(emission, tokens, blank_id=self.blank_id)
processed_lyrics = self.lp.process(text_fn)
# Always compute word-level for the basic LRC output
words = self.lp.get_words_from_path(processed_lyrics, path, frame_duration)
original_lyrics = read_text(text_fn)
lrc = LrcFormatter.words2lrc(words, original_lyrics)
if save:
save_words(words, audio_name)
save_lrc(lrc, audio_name)
# Return syllable tokens for ELRC when requested
if timing_precision == 'syllable':
return_words = self.lp.get_syllables_from_path(processed_lyrics, path, frame_duration)
else:
return_words = words
return return_words, lrc
+56
View File
@@ -0,0 +1,56 @@
import torch
from dataclasses import dataclass
@dataclass
class Point:
token_index: int
time_index: int
prob: float
class Aligner():
@staticmethod
def align(emission, tokens, blank_id=0):
trellis = Aligner.get_trellis(emission, tokens, blank_id=blank_id)
path = Aligner.backtrack(trellis, emission, tokens, blank_id=blank_id)
return path
@staticmethod
def get_trellis(emission, tokens, blank_id=0):
num_frame = emission.size(0)
num_tokens = len(tokens)
trellis = torch.empty((num_frame + 1, num_tokens + 1))
trellis[0, 0] = 0
trellis[1:, 0] = torch.cumsum(emission[:, blank_id], 0)
trellis[0, -num_tokens:] = -float("inf")
trellis[-num_tokens:, 0] = float("inf")
for t in range(num_frame):
trellis[t + 1, 1:] = torch.maximum(
# Score for staying at the same token
trellis[t, 1:] + emission[t, blank_id],
# Score for changing to the next token
trellis[t, :-1] + emission[t, tokens],
)
return trellis
@staticmethod
def backtrack(trellis, emission, tokens, blank_id=0):
j = trellis.size(1) - 1
t_start = torch.argmax(trellis[:, j]).item()
path = []
for t in range(t_start, 0, -1):
stayed = trellis[t - 1, j] + emission[t - 1, blank_id]
changed = trellis[t - 1, j - 1] + emission[t - 1, tokens[j - 1]]
prob = emission[t - 1, tokens[j - 1]
if changed > stayed else 0].exp().item()
path.append(Point(j - 1, t - 1, prob))
if changed > stayed:
j -= 1
if j == 0:
break
else:
raise Exception("Failed")
return path[::-1]
+10
View File
@@ -0,0 +1,10 @@
ORIGINAL_SR = 44100
TARGET_SR = 16000
SEG_HOP_LENGTH = 1
MODELS = {
'en-US': 'facebook/wav2vec2-large-960h-lv60-self',
'en-US-base': 'facebook/wav2vec2-base',
'en-finetuned-base': './model/checkpoint-320000',
'zh-CN': 'jonatasgrosman/wav2vec2-large-xlsr-53-chinese-zh-cn',
}
+84
View File
@@ -0,0 +1,84 @@
import numpy as np
from typing import List
from lsync.lrc_formatter import Word
import dataclasses
from tqdm import tqdm
import os
from lsync import LyricsSync
def get_eval_df(dataset='jamendo'):
import pandas as pd
df = pd.DataFrame(columns=['name', 'audio', 'lyrics', 'annotation'])
if dataset == 'jamendo':
base = "dataset/jamendolyrics"
songs = os.listdir(f'{base}/mp3')
for song in songs:
name, _ = os.path.splitext(song)
df = df.append({
'name': name,
'audio': f"{base}/mp3/{name}.mp3",
'lyrics': f"{base}/lyrics/{name}.raw.txt",
'annotation': f"{base}/annotations/words/{name}.csv",
}, ignore_index=True)
return df
class Eval:
@staticmethod
def eval_all(eval_df, lsync: LyricsSync):
import pandas as pd
record = pd.DataFrame(
columns=['name', 'average_abs_err', 'percentage_of_correct_segments'])
for idx, row in tqdm(eval_df.iterrows()):
try:
words, lrc = lsync.sync(row['audio'], row['lyrics'])
abs_err, perc_corr_segs = Eval.evaluate(
words, row['annotation'])
record = record.append({
'name': row['name'],
'average_abs_err': abs_err,
'percentage_of_correct_segments': perc_corr_segs
}, ignore_index=True)
except Exception as e:
# print(traceback.format_exc())
continue
record.to_csv("eval.csv", index=False)
return record
@staticmethod
def evaluate(words: List[Word], gt_path: str):
import pandas as pd
pred_df = pd.DataFrame([dataclasses.asdict(w) for w in words])
truth_df = Eval.load_ground_truth_jamendo(gt_path)
abs_err = Eval.average_abs_err(
truth_df['start'].values, pred_df['start'].values)
perc_corr_segs = Eval.percentage_of_correct_segments(
truth_df['start'].values, pred_df['start'].values)
return abs_err, perc_corr_segs
@staticmethod
def average_abs_err(ref, pred):
deviations = np.abs(ref - pred)
return np.mean(deviations)
@staticmethod
def percentage_of_correct_segments(ref, pred, window=0.3):
deviations = np.abs(ref - pred)
return np.mean(deviations <= window)
@staticmethod
def load_ground_truth_jamendo(gt_path: str):
import pandas as pd
df = pd.read_csv(gt_path, skiprows=1, names=["start", "line_end"])
return df
@staticmethod
def load_ground_truth_hensen(gt_path: str):
import pandas as pd
data = np.genfromtxt(gt_path, delimiter='\t', dtype=[
('start', float), ('end', float), ('label', 'U10')])
filtered = data[data['label'] != 'PAU']
df = pd.DataFrame(filtered)
return df
+47
View File
@@ -0,0 +1,47 @@
from dataclasses import dataclass
from typing import List
def seconds_to_lrc(seconds, is_word = True):
minutes = int(seconds // 60)
seconds = seconds % 60
hundredths = int((seconds % 1) * 100)
seconds = int(seconds)
formated = f"{minutes:02d}:{seconds:02d}.{hundredths:02d}"
return f"<{formated}>" if is_word else f"[{formated}]"
@dataclass
class Word:
label: str
start: float
end: float
is_word_end: bool = True # False for intra-word syllables (e.g. "Hel" in "Hel-lo")
def __repr__(self):
return f"{seconds_to_lrc(self.start)} {self.label}"
class LrcFormatter():
@staticmethod
def words2lrc(words: List[Word], original_lyrics: str, lang="en-US"):
lrc = ""
counter = 0
word_end = None
for line in original_lyrics.splitlines():
if line == '': continue
if word_end:
lrc += f"\n{seconds_to_lrc(word_end, False)}"
else:
lrc += "[00:00.00]"
if lang == "en-US":
splitted_words = line.split(' ')
elif lang == "zh-CN":
splitted_words = line
for original_word in splitted_words:
if original_word == '': continue
word = words[counter]
word.label = original_word
lrc += f" {word}"
word_end = word.end
counter+=1
return lrc
+154
View File
@@ -0,0 +1,154 @@
from dataclasses import dataclass
from .util import read_text
from .lrc_formatter import Word
SEPARATOR = '|'
@dataclass
class Segment:
label: str
start: int
end: int
class LyricsProcessor():
def __init__(self, lang: str = "en-US") -> None:
self.lang = lang
def process(self, text_path):
text = read_text(text_path)
if self.lang == "en-US":
return self.__process_en(text)
elif self.lang == 'zh-CN':
return self.__process_cn(text)
elif self.lang.startswith("en") and self.lang.endswith("-base"):
return self.__process_en(text, is_upper=False)
def __process_en(self, text: str, is_upper=True):
# preprocessing
if is_upper:
text = text.upper()
else:
text = text.lower()
text = text.replace(' ', '|')
text = text.replace('\n', '|')
text = text.replace('_', '\'')
text = text.replace('’', '\'')
return text
def __process_cn(self, text):
# preprocessing
return text
def get_words_from_path(self, text, path, frame_duration):
# Skip repeating char
i1, i2 = 0, 0
segments = []
while i1 < len(path):
while i2 < len(path) and path[i1].token_index == path[i2].token_index:
i2 += 1
segments.append(
Segment(
text[path[i1].token_index],
path[i1].time_index,
path[i2 - 1].time_index + 1
)
)
i1 = i2
if self.lang.startswith("en"):
return self.__merge_en(segments, frame_duration)
elif self.lang == 'zh-CN':
return [Word(s.label, s.start * frame_duration, s.end * frame_duration) for s in segments]
def get_syllables_from_path(self, text, path, frame_duration):
"""Like get_words_from_path but returns syllable-level Word objects.
Intra-word syllables have is_word_end=False; the final syllable of each
word has is_word_end=True. Falls back to word-level if pyphen is not
installed.
"""
# Deduplicate path entries (same logic as get_words_from_path)
i1, i2 = 0, 0
segments = []
while i1 < len(path):
while i2 < len(path) and path[i1].token_index == path[i2].token_index:
i2 += 1
segments.append(
Segment(
text[path[i1].token_index],
path[i1].time_index,
path[i2 - 1].time_index + 1
)
)
i1 = i2
if self.lang.startswith("en"):
return self.__syllabify_en(segments, frame_duration)
elif self.lang == 'zh-CN':
return [Word(s.label, s.start * frame_duration, s.end * frame_duration) for s in segments]
def __syllabify_en(self, segments, frame_duration, separator=SEPARATOR):
"""Split word character groups into syllable-level Word objects via pyphen."""
try:
import pyphen
dic = pyphen.Pyphen(lang='en_US')
except ImportError:
# Graceful fallback: return word-level alignment
return self.__merge_en(segments, frame_duration, separator)
syllables = []
i1, i2 = 0, 0
while i1 < len(segments):
if i2 >= len(segments) or segments[i2].label == separator:
if i1 != i2:
word_segs = segments[i1:i2]
word_text = "".join([s.label for s in word_segs]) # uppercase
# pyphen works on lowercase; character count is identical
positions = dic.positions(word_text.lower())
if not positions:
# Single-syllable or unrecognised word — emit as one token
syllables.append(Word(
word_text,
word_segs[0].start * frame_duration,
word_segs[-1].end * frame_duration,
is_word_end=True,
))
else:
boundaries = [0] + list(positions) + [len(word_text)]
for j in range(len(boundaries) - 1):
s_char = boundaries[j]
e_char = boundaries[j + 1]
syl_segs = word_segs[s_char:e_char]
if not syl_segs:
continue
is_last = (j == len(boundaries) - 2)
syllables.append(Word(
word_text[s_char:e_char],
syl_segs[0].start * frame_duration,
syl_segs[-1].end * frame_duration,
is_word_end=is_last,
))
i1 = i2 + 1
i2 = i1
else:
i2 += 1
return syllables
def __merge_en(self, segments, frame_duration, separator=SEPARATOR):
# Merge chars to word
words = []
i1, i2 = 0, 0
while i1 < len(segments):
if i2 >= len(segments) or segments[i2].label == separator:
if i1 != i2:
segs = segments[i1:i2]
word = "".join([seg.label for seg in segs])
words.append(Word(
word, segments[i1].start * frame_duration, segments[i2 - 1].end * frame_duration))
i1 = i2 + 1
i2 = i1
else:
i2 += 1
return words
+73
View File
@@ -0,0 +1,73 @@
# Import directly from the concrete submodule to bypass transformers' lazy-loader,
# which raises a misleading "Are this object's requirements defined correctly?" error
# when the backing module can't be found via the top-level package.
from transformers.models.wav2vec2.modeling_wav2vec2 import Wav2Vec2ForCTC
from transformers.models.wav2vec2.processing_wav2vec2 import Wav2Vec2Processor
from transformers.models.wav2vec2.tokenization_wav2vec2 import Wav2Vec2CTCTokenizer
from transformers.models.wav2vec2.feature_extraction_wav2vec2 import Wav2Vec2FeatureExtractor
from transformers.utils import logging
import torch
from .config import TARGET_SR, MODELS
from .lyrics_processor import LyricsProcessor
from .util import get_audio_segments
logging.set_verbosity_warning()
class PhonemeRecognizer():
def __init__(self, lang="en-US", device: str = None) -> None:
model_id = MODELS[lang]
if lang.endswith('-base'):
tokenizer = Wav2Vec2CTCTokenizer(
"vocab.json", unk_token="[UNK]", pad_token="[PAD]", word_delimiter_token="|")
feature_extractor = Wav2Vec2FeatureExtractor(
feature_size=1, sampling_rate=16000, padding_value=0.0, do_normalize=True, return_attention_mask=False)
self.processor = Wav2Vec2Processor(
feature_extractor=feature_extractor, tokenizer=tokenizer)
else:
self.processor = Wav2Vec2Processor.from_pretrained(model_id)
if device is not None:
self.device = device
else:
self.device = "cuda" if torch.cuda.is_available() else "cpu"
self.model = Wav2Vec2ForCTC.from_pretrained(model_id).to(self.device)
self.lp = LyricsProcessor(lang=lang)
self.lang = lang
def recognize(self, vocals):
self.duration_sec = 0
segs = get_audio_segments(vocals)
logits = self.__recognize(segs[0])
for seg in segs[1:]:
logits_seg = self.__recognize(seg)
logits = torch.cat((logits, logits_seg), dim=1)
# normalize the emission with log_softmax() to avoid numerical instability by computing prob in log-domain
emission = torch.log_softmax(logits, dim=-1)[0].cpu().detach()
pred = self.get_pred(logits)
transcription = self.get_transcription(pred)
# get real time
frame_duration = self.model.config.inputs_to_logits_ratio / TARGET_SR
return emission, pred, transcription, frame_duration
def tokenize(self, text_path):
text = self.lp.process(text_path)
return self.processor.tokenizer(text).input_ids
def __recognize(self, vocals):
vals = self.processor(vocals, return_tensors="pt",
padding="longest", sampling_rate=16000).input_values.to(self.device)
self.duration_sec += vals.shape[1] / TARGET_SR
with torch.no_grad():
logits = self.model(vals).logits
return logits.cpu().detach()
def get_labels(self):
return [k for k, v in sorted(self.processor.tokenizer.get_vocab().items(), key=lambda x: x[1])]
def get_pred(self, logits):
return torch.argmax(logits, dim=-1)
def get_transcription(self, pred):
return self.processor.batch_decode(pred)
+55
View File
@@ -0,0 +1,55 @@
import librosa
import soundfile as sf
from .config import ORIGINAL_SR, TARGET_SR
from lsync.lrc_formatter import Word
from typing import List
import dataclasses
import numpy as np
window_size = int(TARGET_SR * 15)
hop_length = window_size
def get_audio_segments(audio):
"""Split audio to segments"""
return librosa.util.frame(audio, frame_length=window_size, hop_length=hop_length, axis=0)
def get_audio_segments_by_onsets(audio):
onset_times = librosa.onset.onset_detect(
y=audio, sr=TARGET_SR, backtrack=True)
onset_boundaries = np.concatenate([onset_times, [len(audio)]])
segments = []
start_onset = 0
for onset in onset_boundaries:
segments.append(audio[start_onset:onset])
return segments
def read_text(text_path):
with open(text_path, 'r') as file:
data = file.read()
return data
def save_audio(audio, name, sr=ORIGINAL_SR):
sf.write(f'output/vocals/{name}.wav', audio, sr)
def save_audio(audio, name, sr=ORIGINAL_SR, out_path="output/vocals"):
sf.write(f'{out_path}/{name}.wav', audio, sr)
def save_audio_file(audio, path, sr=ORIGINAL_SR):
sf.write(f'{path}.wav', audio, sr)
def save_lrc(lrc: str, name: str):
with open(f'output/lrc/{name}.lrc', 'w+') as fp:
fp.write(lrc)
def save_words(words: List[Word], name: str):
import pandas as pd
df = pd.DataFrame([dataclasses.asdict(w) for w in words])
df.to_csv(f"output/words/{name}.csv", index=False)
+43
View File
@@ -0,0 +1,43 @@
import torch
import numpy as np
from demucs.pretrained import get_model
from demucs.apply import apply_model
from demucs.separate import load_track
import librosa
from .config import TARGET_SR, ORIGINAL_SR
class VoiceExtractor():
def __init__(self) -> None:
self.device = "cuda" if torch.cuda.is_available() else "cpu"
self.model = self.__load_model()
self.vocals_source_idx = self.model.sources.index("vocals")
self.sr = self.model.samplerate
def extract_voice(self, audio_fn, post_process=True):
audio = self.__load_audio(audio_fn)
vocals = self.__extract_voice(audio)
return vocals if not post_process else self.post_process(vocals)
def post_process(self, audio, sr=ORIGINAL_SR, target_sr=TARGET_SR):
audio = librosa.resample(audio, orig_sr=sr, target_sr=target_sr)
return audio
def __extract_voice(self, audio: np.array, shifts: int = 1) -> np.array:
"""Extract mono vocals from the audio"""
ref = audio.mean(0)
audio = (audio - ref.mean()) / ref.std()
with torch.no_grad():
sources = apply_model(
self.model, audio[None], device=self.device, shifts=shifts, split=True, overlap=0.25, progress=False)
vocals = sources[0][self.vocals_source_idx]
return vocals.cpu().numpy()[0, ...]
def __load_model(self):
model = get_model(name="htdemucs", repo=None)
model.to(self.device)
model.eval()
return model
def __load_audio(self, audio_fn: str) -> np.array:
return load_track(audio_fn, 2, self.sr)
+843
View File
@@ -0,0 +1,843 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Lyrics-audio Alignment"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"audio_name = \"Kinematic_-_Peyote\"\n",
"lyrics = f\"dataset/jamendolyrics/lyrics/{audio_name}.raw.txt\"\n",
"audio = f\"dataset/jamendolyrics/mp3/{audio_name}.mp3\"\n",
"lang = \"en-US\"\n",
"blank_id = 0"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import torch\n",
"torch.cuda.is_available()"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" <audio controls=\"controls\" >\n",
" <source src=\"data:audio/mpeg;base64,SUQzBAAAAAB7N1RJVDIAAAAIAAAAUGV5b3RlAFRQRTEAAAALAAAAS2luZW1hdGljAFRSQ0sAAAADAAAANQBUQUxCAAAABwAAAEtpdGVzAFREUkMAAAAGAAAAMjAxNwBUQ09OAAAABgAAAFJvY2sAVENPTQAAAAsAAABLaW5lbWF0aWMAV1BVQgAAABcAAGh0dHA6Ly93d3cuamFtZW5kby5jb20AVFBVQgAAABgAAABodHRwOi8vd3d3LmphbWVuZG8uY29tAFRYWFgAAAAiAAAAVGFnZ2luZyB0aW1lADIwMTgtMDYtMjFUMjA6MjE6NDgAVEVOQwAAACYAAABKYW1lbmRvOmh0dHA6Ly93d3cuamFtZW5kby5jb218IExBTUUAV09BUwAAACcAAGh0dHA6Ly93d3cuamFtZW5kby5jb20vZW4vYWxidW0vMTY3ODE0AENPTU0AAAAoAAAAZW5nAGh0dHA6Ly93d3cuamFtZW5kby5jb20gY2Nfc3RhbmRhcmQAV09BRgAAACgAAGh0dHA6Ly93d3cuamFtZW5kby5jb20vZW4vdHJhY2svMTQ0MjAzMABXT0FSAAAAKAAAaHR0cDovL3d3dy5qYW1lbmRvLmNvbS9lbi9hcnRpc3QvNDU3NTczAFdDT1AAAAAvAABodHRwOi8vY3JlYXRpdmVjb21tb25zLm9yZy9saWNlbnNlcy9ieS1uZC8zLjAvAFRDT1AAAAAwAAAAaHR0cDovL2NyZWF0aXZlY29tbW9ucy5vcmcvbGljZW5zZXMvYnktbmQvMy4wLwBBUElDAAB1fgAAAGltYWdlL2pwZWcAAwD/2P/gABBKRklGAAEBAQBgAGAAAP/bAEMACAYGBwYFCAcHBwkJCAoMFA0MCwsMGRITDxQdGh8eHRocHCAkLicgIiwjHBwoNyksMDE0NDQfJzk9ODI8LjM0Mv/bAEMBCQkJDAsMGA0NGDIhHCEyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMv/AABEIAZABkAMBIgACEQEDEQH/xAAcAAACAgMBAQAAAAAAAAAAAAABAgADBAUGBwj/xABIEAABAwIFAgQCBwQIBAQHAAABAAIRAyEEBRIxQVFhBiJxgRORBxQyobHB0RUjQmIXJUNSY3KS4TNTgoMWJPDxVHN0hJOisv/EABoBAAMBAQEBAAAAAAAAAAAAAAABAgMEBQb/xAAzEQACAgEDAwIEBAYDAQEAAAAAAQIRAwQhMRJBUQVhEyKRoRQVQoEyUnGxwdEjM/Biov/aAAwDAQACEQMRAD8A8yJDLJdW43lR5JMSgD2C7jwewwsPKLp/iyRaAgNkIlAuS0Xi0IGEKck/aHojBJtsmiO47eLoEeaYsiAQgTvf0QLuBwBbBF+qpezlWu2F/VVumN0i4sqhEkhWN0gXSnS50C6C7YnG6bVAtdCQNht1UkaUDJqJP6KTKEwUOUDocNcidVhEfmlk8JgHG5meEEjtb5Z+aIA4sowEtMIxFt0ENg/BAt5tdPYJLh1kAmAE9kTGlQDnZBw6EwgYwkC1yiJFkAJ3KMbJk2QiN0pIA2TgXQIkQgExAb7WRceQiGjdAn2CQwAmZ5UjVY7og9d0YgkzEoAUtiCjFgU5uEoFoTCxS0E2FkSNJBiyYBE+ZqQWKY0Seqnw4EjhQjyxdWOEQBwExWBrYueiLqWzjuUW+Z0X9FaBNNogWO6pIhyaKHUxGkm25umDR/CgQC8oxcQkO3QNJDtkHiYBAurIkzKrIbr3QCYCIQcLC3omcLmEhNkikwNBmE5MEBIHFplM2SZKAYxMEogjV2SmwKkdSgRW0DVcz7Ji3eEDY3TDZIpgi6CsMgdVUQSbIEhqcB2ytkA8KunaQd08C6YpckLgUknpZORxKUNSBUCeigjpsrdIaEpgGEwsVzGkExdVwACQrHA3KTSQOqRSZVCEWVpaCO6GmAeqRXUVIjdMGydlYGEOm3umNyFYQEdZLy6VZpA2CmjfoUEOSIAAZ45hEAQEWiLIgAEi1kybF5uFIsbhQdUwaCZKBCgTMD1UAncwoYaRGx6JtIInjqgYhEcJ2tQiUzZBN9+iYmwabygWp4UOyBWLBSObq5Tg8IcbIGKIFpCVWbJHCDdIaBBTRYcIgR6qE9v9kICAQ2N0SI/JRv3p3AFMTe5GtBAlQgTA3RYYHoo6OEE9wRperHODaZHVIHRwpMwCqukIAEBGwko8oOjZSFgAO/ukIm6scbQgBHCBplUuJCBadlaRCERukVZVG9rpg4DhQXJCZwGmOUDsrmUwEndLpINkzR5ikMUwSLInUBNkQJKaQAgLFg3B3QY3k2TH7U8oxKKFZA0CyzqOW1sRlmLx7DTFLDFgeCbnUYELEa2Aup8N0sLV8OZyMbWNLDfEoGo5okwCTA7nZcutzywYuuPmP3aTLwwWSdPw/wCxom5VX/ZDszqOZToa9FMPJ1VTzpHIHVJj8BVy7EMo1H0XudTbUBpP1CCNp6rbeLvrAzVrHFgwQpA4IU/sClxHfr3W1ZlGXnxfg8IcKwYapgQ97BtqNMku9ZuuRa9xhHLPhqUqXhVtfn/Js8CcnCPZpfU4yDaVCNXAXU5b+zM4disubllHD6cO+pQrsLjUBaJ8xJvKmXZO+nkOHx1DL6GNxOKc6PrDwGUmAxYEiSTPotZeoxhanGnaVOu9tb3XZ/QhaZy3i7RyhaJQHNl2zckwZ8R5N8TCUqbcY13x8K1+trHgHaDsbGJVGVtyjNMTi8tfljKNKlSfUp16bj8bybyTYyO1lD9Uh09Si2krfGytrzvx2K/Cu6b9v8nGOBJsEul5XWUzgc4yTMyzLKGErYKm2tSfRLpc2YLXSb+qOTYau/DYd1TIsvdhHO0vxGKqaHVb3LSXD7grlr4xjJyjTi6abXi/IRwttJPk5ejh61cltKlUeWguIY0mANz6LKwuAq4nB4vEtqUmtwzA9we+HOBMeUcrscppYfKfEGe5fTw9KoylQqua+pJdpgHQb7XvytPhWYTMMuz/ABjsHh6bqNCk6k2mDppnVBIk8rL8wcrcVUfkp7fqfiyvgJUm99/sc5JjZSeJXSYxuC8PYTB0jl2HxmOxFAV6tTEyWsDtmtaCPmudr1mV8Q+qyhToNe6RTpzpb2E3XZg1LzfNGPy9n5Mp4lDl7mxqZLiWeG6OeF9L6rVxLsM1snWHATJG0LIq+Gcbhsvy3F4qrhsO7MXhuHoVHxULCY+IRENb3JXb+E6GSVvAOUOz/Etp4NmdPIYWkiq8tAaD0beT2C5TxCzNKvj91DPQHYg4unTc0CGGmXANDR/c0my2Um3RcsSUU/NGrxmSYzC5risuY0YqthidZws1GwBJdI4A5WBt7r1jJXYLJ/HHjDLsNlWD+BQwmIfTkOkMDG/u9/snnlaHw9lNLPsFmniGlkeCqPoPp0MNlzanw8PrIlz3FzpIAvE3JQsgnp7ez8/Y4Ijj3Uiy9Az3IMN+wsHmmZZfhMoxbMczD4mlgqrSypRd/aBoc6HC4SeNcD+z6FRmD8PZczJ3vb9SzTDS9zm/zPDiJN5BHomp2TLC4pvwcDEJwo5tpCUb+i0MHuPwg7ZTUOVNQ9UEoBiEo36IzuYQBvsky0GPVI4EBMQevslkmyBogsmdNo5SbutCd02B4QgfIdgAd+iDS7so0TcqwAIJboMRBBULZTQIuEQJ9k6IsQiNio25UfIlM0QEx9iEWQA9EdwoNrpCEIkwpF7JuUvKBoDrlITZWTYlVgE3QUgQRdMATfhAklFriAEhskBu/soG2Qc4ufKstAQJ7CRA5SgyfwTmI3QtEpjQHNM7o3tAurGN1kBQiOEiersKNWxW0weZUsPkGZYFzXmrinUnMLQIGkyZWrdIhOGrPJijlXTLs0/o7X9hxm4O0bVuZ0a3h9+W4xlR1SideDqtAOgn7TTP8JWyZ4iwg8R4bMfhVvg0sKKLmwNWoMLbX2lcyR3UJ6Lmn6fhld97/wD1z9a+potTNV7V9uDZZHmNHK8wqYis17muoVKY0ATLhAT4XMcvxGU0cszVmIaMO5zqGIoAEtDt2kHcStXIHYpHhpBNtrK8mjxzk57p7brtV/7YoZ5RVdjdZdjcsyrP8Hi8N9aqYajd5qNaHkwRYAxFwq8nzOhl+Y4nEVWVHNqUatMBgEy4WWlY/Q2CDZQVQTMQOqUtJjkpKVu1T/pv/spZZpquzs22V4+lgsBmlCo15di8MKLC0CAdU37LYYjM8ox1PAV8WzFDEYOiyl8Cnp+G/SbGf4Z5submCDIuniyWTRY5z63ad9n7V/YUdROKrt/5nRDP8H/4qx2YupVjhcZTdSe0QHgOaBI44WHQzDL8Dgs4weGOIqUsXSYyi6o0AyDJ1AH8FpnBANUr0/EtldVFc/y7ov8AETe79/vyb6pmmU5rgcNSzZuKpYnDU/hNr4YNcKjBsHA8jqtDiRhxiKn1X4nwNXk+LGqO8WlAtEmyGmYWuHTRxN9DdeOy/oTLK5rc31TO8PU8B4bIxTq/WaWPfinPIGgtLYjeZ9lm1vFGCzLKsoOY065zfLKzGtxDACK2Ha4HS6TOocH9VyxENSkQFv0oFlZ29Hxhl7PH+bZxUoYh2W5lTq0HsaAKrWPaBMTEiNpWHgM5yHBU8zyOtTx2LyDGOZUZUIa3EUqjRZ4H2T0jouWbAR0xexS6EP48jcY1nhelUw4y+rmtZvxQa7q1KnT/AHfRkE+bubLcVc6yPK/DOZ5Vkz8zxJzLQH/XGsYyiGmZAaTLuJXGwoJnfdPp8k/EatpD7hAjlEIkiFZiLpBRIhSegTdgUAVmwlANlMSNlARB4SGuCOASECY1bp3zaOUGsA3QNbIgZB4UO/UommSJC6nJvA2NznLqeOGLoUG1AS1r2uLine truncated
" Your browser does not support the audio element.\n",
" </audio>\n",
" "
],
"text/plain": [
"<IPython.lib.display.Audio object>"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from IPython.lib.display import Audio\n",
"from lsync.config import ORIGINAL_SR\n",
"\n",
"def show_audio(audio, sr):\n",
" return Audio(audio, rate=sr, autoplay=False)\n",
"\n",
"show_audio(audio, ORIGINAL_SR)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Vocals Extraction"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"from lsync.voice_extractor import VoiceExtractor\n",
"\n",
"ve = VoiceExtractor()\n",
"vocals = ve.extract_voice(audio, post_process=False)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Extracted Vocals"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" <audio controls=\"controls\" >\n",
" <source src=\"data:audio/wav;base64,UklGRvzJ9gBXQVZFZm10IBAAAAABAAEARKwAAIhYAQACABAAZGF0YdjJ9gDm//b/9v/1//L/9v/z//v/7//u/+//8P/l/+//7v/0//L/9v/w//T/9P8AAPr/+f/s//L/+//z/9r/9P/v/+H/3f/s/9//4v/o/+j/5//h/9//4f/Y//L/7//v/+//+f/o/+v/9v/+//X/9P/l/9v/1P/v/+r/6v/w/+7/2//e/+X/9P/y//n/4//t/+r/9f/u/+b/2//p/+z/8v/y/+r/2P/t/97/7//z/+//4P/n/+z//P/2/+3/5v/t//L/9f/1//z/8//2//H/CwD9/w0AAAD1//P/+f8BAAUA+P/4/+n/9f/i/+z/9v/7/+3/6P/i//j/6v/5/+7/7//r/+r/4//u/+b/8//t//n/+v8BAPj////q/+r/8/8JABMACwABAAwA7//7/wkACQADAAoAAAD//+7/9f/t/+3/8v8BAPr//P/z/wAA/f/0/+7/+f/p//r/7f/0/+//AAD8/wEA+P/9/wQACAAHAAsAAgAUAAoAGQAWABQABwAIAA8AFgATAA8AEQASAAoAGAATABMADgAaABQAHQAVABAAEgATAAkAEAAMABcABgAKAAoAEQALABAAAAD//wYABgABAAwACgALABAAEwASABoAEwAeABcAGQAbABUABwAMAAgADAAKACEAHgAcABYAGAAMABMAEgAeABAADwAHAA0ACwAOAAoADgATABEACwAPAAAA9//x/wIA9P/u//H/7v/h/+H/4P/p/+j/8v/j/93/1f/V/9b/0f/N/8z/zf/Z/9P/1P/U/9P/0v/k/9//7P/d/9f/0v/c/9z/0v/N/8P/zP/R/9T/1f/S/9j/0v/R/9b/3P/H/8P/wf/A/8H/xP/C/7z/uf+//8D/yf/J/8T/vf/F/7v/vP+0/8D/sP+u/7X/tP+r/7H/q/+t/6b/rP+l/7H/rP+o/6P/sf+q/63/rP+8/7D/sP+u/7X/qf+p/6v/p/+g/6v/nv+b/47/l/+Q/6X/mv+T/4T/ov+b/6H/kP+E/3n/f/91/33/ZP9v/13/Uf9O/1j/V/9U/1z/Qf9I/yr/Df8Y/+P+8v6Q/qj+of6Z/mH/cgBQ/mf/if8YAEkAfQC4AK4A2QCgAKsAggCCAG4AcgB2AH8AfAB6AH8AjgCJAJIAkgCSAIIAcQBkAG8AagB2AGoAZgBgAHcAfwCAAH8AgACEAIcAhQCEAIEAfwBxAHgAegBxAHAAbQB3AHQAegB4AIIAcQB7AHQAdgBzAHkAdgB3AHkAewBwAIQAZgBfAF0AWgBKAD0APQA9ADAAOwA2AEEANAA2ACoAKgAwAD4AOgA9ADkAMAA4AEoANgA+ADoAQAA5AEEAQQBLADkAPAAtADMAMQAvABwAIgAfACMAIAAkABUAHgAWACAAHgAlAB4AJQAbACcAIgAdABMAEQAYAB4AFwAZABMACwAGAA4A+P8AAAEACwD2//T/8f/y/+X/9//n/+z/4v/y/+v/9f/l/+7/8P8AAPj//f/y//n/6f/6//r/BgD2/wAA9/8AAOz////z//r/7/8AAPL/AgD4/wsA+f8HAPz/BAD1/woA9//7/+3/+//s////5P/z/9z/6P/l/+z/2f/s/9z/6v/j//X/6f/1/+T//P/t//7/+f8CAPD//P/8/wIA8P/4/+j/+//t//n/7v/9/+z/9P/0/wMA8//2//H/AAD7/wYA9v8EAPn/AAD6/wAA9//5//D/+P/p/+//8////+T/8P/m/+r/6P/u//T/7P/v//z/3v/f/9P/yf/E/+L/4/8EAPL/+//t//3/9f8CAAMADAADAA8AAgAPAPr/CgD6/wcA+/8CAAAABADs//f/7P8CAPX//P/x//n/5v/s/97/6v/k/+7/1//d/9T/4f/g/+7/5P/r/+D/7v/j//T/7f/x/9z/6P/p//n/7f/w/9//9f/x//X/9v/5/+z/+v/2/wAA/P8BAPD/9//w//v/8f/z/+b/8//y////+v/4//L/+//x//3//P8FAPT/9//r/+7/7v8BAPH/+f/y//z/7v/8/wIAAQD5/wMA9//+//v/AwABAAQAAAAMAPr/CAAKAAQA/P8GAAcAAwD5/wEA/P8CAPr/AgD5/wEA/v8DAPv/AQD6/wMA+P8AAPj/BAD+/wYA+f8FAAUADQAAAAwABgASAAMADgAEABEABQARAAIADAABAAsAAAALAP3/BAD4//7/8P8FAP3/AgD3/wwA+v8FAP7/CAAAAAoA/v8FAP7/DQD9/wYA//8KAP//CgAFABIABgANAAQADAAAAAIA/f8EAP7/BgD0//3/+/8DAPn/AQD5/wAA+/8FAP7/BAD7/wYA/P8EAP7//v/2/wEA9P/8/+///P/4/wMA/P8BAPD/9P/z//3/8/8AAPj//v/5/wYA9v8EAAUACgD9/wgAAQAJAPz/BAD4/wcA+/8KAP//CAD5/wsADQAUAAsACwD//wsA//8HAPj/AwDz////9//8//D/AwD9/wYA+P8FAPn/AgD0/wMA9P8HAAAACwD5/wEA+/8FAP3/BQD5/wEA8f8CAPf/AAD1//7/9v8CAPb////3/wAA+P8CAPb/AQDv////9/8AAPD/+//z//z/9P/6//L//f/1/wAA9P8DAPr/BgD5/wUA//8BAPn/BgD//wcA/v8GAP3/BgD8/wUA/P8KAP//BgAAAAkAAAAJAP3/BQD5/wUA+f8DAPr/AAD4/wAA+f8FAPv/BgD7/wIA+f8FAPT/BAAAAAcA+/8GAP3/CwD//wkA/v8JAP7/CAD7/wgA+v8HAPz/BQD2/wQA+v8EAPf/AgD4/wcA9/8GAAAACgD6/w4A//8NAP3/DAD+/wkA+P8IAPX/BAD+/wsA/f8IAAAACgD6/woA/f8KAPv/BQD1/wQA9/8GAPr/CQD6/wYA+v8CAPv/BwD4/wIA9/8FAPX/AAD1/wEA+P8EAPr/BgDz/wUA/P8HAPz/BwD7/wMA+f8GAPz/BgD6/wYA+f8FAP3/DAD+/wkA9/8GAPr/AwD5/wMA9f8AAPn/BgD4/wMA8f8AAPP/AADy//v/8v/8//H//P/u//r/7f/4/+//+//s//n/6f/3/+7/9v/t//n/7//9//P//P/4/wIA9f8CAOb/9f8DAPv/7v/5/+z/9//u/wEA9f8AAPP/AADy//n/8//8//D//P/0//7/8v8AAPT////1/wMA9/8BAPb/AAD0/wIA9v/+//T//v/y/wAA9/8CAPb/AQDy/wEA9v8AAPb/AwD1/wEA+P8CAPf/AgD6/wUA+f8EAPf/BAD5/wYA+P8AAPr/AgD2/wEA+f8CAPj/AQD1/wEA9/8DAPX/AQD4/wIA+f8AAPb//f/w////8/////j/AwD1/wMA+f8AAPb/AADy/wAA8/8AAPX/AAD3/wMA+v8HAPz/BQD4/wIA+P8CAPT/AADx/wAA8/////X/AAD2/wEA+f8EAPX/AAD0/wEA9v8BAPT/AwD5/wYA+v8FAPj/BgD8/wYA9/8GAPb/AwD0/wEA8P/5/+r/+v/u/wAA8P////D/AADw//v/8P8AAO///v/y//7/8f8CAPT/BAD4/wcA8/8GAPf/AwDy/wMA9P////T/AADs//z/8f8AAPD/AAD0/wgA+P8IAAEABQD7/wkA/v8KAP7/EQAAAA0A/v8SAP7/CwD+/wsA+/8HAPj/CAD1/wgA+f8HAPr/DgD8/w0A//8PAPn/BQD1/wIA8f8JAOr/+v/v/wAA9f8DAPj/BwD4/xIA8f8bAPr/DwD2/wMA7f8FAPP/AAAAAA8ADAArABgARQAiACkACQAhAPf/0//T/6H/mf+f/5P/t/+u/8//zf/o/93/9//w//v/7//z/+j/+//s////7v/9//D//f/z/wIA/v8BAPf/DQD7/wQA/v8KAPr/CAD8/woA+P8LAPz/CQD//wwAAAAMAAAACwD9/woA/v8GAPj/CQD5/wQA9/8IAPn/BgD1/woA+/8CAPb/BgD0/wcA+/8LAPv/CwAAAA0A+v8JAPr/BQD3/wcA+P8IAPr/CgD2/wQA+P8FAPf/BwD4/wQA+P8HAPX/CAD4/wQA+P8HAPr/BgD6/wgA//8OAP//CQD5/woA+v8HAPn/CQD4/wYA+/8IAPr/BQD2/wQA+f8EAPj/CAD5/wYA/P8IAPr/BQD7/wcA+v8IAPv/CAD+/wkA/v8LAPv/BwD6/wYA+P8DAPX/BQD2/wYA+P8EAPn/BAD7/woA+v8HAPj/BAD5/wgA9v8HAPr/CAD9/w0A/f8LAPv/CAD6/woA+/8LAPr/CQD7/wsA/f8KAP3/CwAAAA4A/f8PAP7/CQD6/woA+/8KAPv/CAD5/wcA/P8JAPr/CQD6/wcA9/8IAPn/AwD7/wcA+v8GAPn/BgD5/woA+f8JAPv/BAD7/wUA+v8EAPb/AAD1/wIA9/8EAPf/CAD4/wMA+v8EAPn/BgD5/wIA9/8EAPj/AgD3/wYA+f8DAPb/AwDs//X/AAADAPb//v/v//z/8P/9//D/AgDz/wMA9/8BAPr/BAD4/wQA+P8FAPr/CQD5/wkA+P8IAPn/CgD9/wYA+v8HAPr/BwD2/wYA9v8HAPv/CgD6/wkA+v8IAPv/CAD3/wcA9v8IAPr/CQD6/wgA+/8IAPr/CAD6/wYA+P8FAPb/AwD7/wkA+f8IAPn/BwD6/wkA//8LAP3/DAD//wsA/P8LAP7/CAD5/wUA9/8EAPj/BgD5/wgA+v8IAPv/CAD8/wkA+f8JAPv/CAD7/wkA/v8JAP3/DAD7/wgA+v8GAPf/AwD4/wYA9v8GAPb/BwD4/wkA+/8JAP7/CgD+/wwA+/8LAP3/CQD6/wkA+v8HAPn/BwD6/wYA+P8IAPn/BwD3/wUA+/8HAPfLine truncated
" Your browser does not support the audio element.\n",
" </audio>\n",
" "
],
"text/plain": [
"<IPython.lib.display.Audio object>"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from lsync.config import ORIGINAL_SR, TARGET_SR\n",
"from lsync.util import save_audio\n",
"\n",
"save_audio(vocals, audio_name)\n",
"show_audio(vocals, ORIGINAL_SR)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Vocals Post Processing "
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" <audio controls=\"controls\" >\n",
" <source src=\"data:audio/wav;base64,UklGRtCJWQBXQVZFZm10IBAAAAABAAEAgD4AAAB9AAACABAAZGF0YayJWQD0//P/9f/y/+v/7//y//X/+v/y/+3/5//h/+b/4//e/+//7//x//j/2f/s/+f/4v/z/+v/7f/h//L/4f/s/+j/7v/x/+v/+//z/wgA+f/8//3/6P/0/+z/7f/z/+f/6v/x/wAA7v/+/w4A+f8IAAMA8v/w//z/+v/1//D/8v/8//3/BwAJABYADwAPABMAEAAUABUAGAAOABAACwAOAAYAAgAKAA8AFwAZABkACAARAB8AEgAWABEACQAPABAAAwD0//X/4P/o/+b/1v/O/8//1P/T/+T/2f/X/8r/zv/V/9L/0P+8/8L/uf/D/8P/uv+3/63/sP+l/6v/pf+q/6z/tP+o/6n/nv+V/5T/lP+W/4n/cP9t/0z/U/9M/xr/8f57/jL/T//t/8cArQCVAGoAiwB+AKEAeQBzAGsAcgCDAIcAiQB9AHgAcwB9AHwAeQB5AH0AeQBlAEwAOQA7ADgALQA/ADcAQQA7AEMAQQA1ACoAIAAiABkAIQAjACIAFgAZABUAAgADAPr/7v/q/+v/7P/z//v/8f/8//3/9v/4//b/AQAAAAEA+f/1/+7/5P/l/+L/7f/t//f/+//7//X/8f/0//T/+f/3/wAA/P/+//T/7//z/+j/7P/x/+n/x//j//r/9P8FAAkABgAAAAEA+f/0//v/6//m/+L/1//p/+T/7v/n/+v/7//t//f/8/8AAPT/9f/t//j/9//2/wAA7v/1//b/9v8AAPz///8DAAQABQADAAAA/v//////AAD8////AAAFAAkACgAKAAsABgAGAP7/+v8AAAAAAwADAAMAAgAEAAkADAADAAEA/v/9//7//v8BAAAA///7//b//P/9//L/+//7/wAABAAFAAAAAAADAAQAEQAGAAIA/P/4//z/AAD9//v/AwD//wAA/v/7//v/+//8//3/+//5//j/9//3//n//v8AAAAAAAADAAAAAgADAAUAAgD///7//P8AAAAA/f8AAAEAAwAEAAIAAQAAAP///v/+/wAABAAFAAQAAAAAAAQAAwACAAIA/f8AAAAA//////7/+//9//////8DAP//AQAAAAEABAAAAP//+/////v/+f/3//b/9P/z//T/8P/z//T//P/5//b/+f/w//n/+v/3//b/9//5//r//f/7//z/+P/7//z/+v/8//z//f/+/////v/+//z//v/8//z//P/9//z/9v/7//3//f/6//r/+////wAA/v/7//n/+f/8//3/+//7//7/AAAAAAAA/v/6//L/9f/5//b/9//3//n//f////3//P/5//b/+P/8/wIAAgADAAcABwAGAAIAAAAAAAEABgAEAP3/+v/0//z/AAAFAAYA+v/7/wYAJwAsAA0Ayv+U/63/0f/v//T/8P/2//X//P8AAAEAAgACAAIAAwAGAAUAAwAAAAAA//8AAP7//v8CAAQAAwD//wAAAAD+////////////AAAAAAUAAwAAAAAAAAAAAP3/AAAAAAAAAAABAAIAAwABAP7//f//////AQAAAAAAAAADAAQAAQACAAEAAwADAAcABAACAAEAAAABAAAAAAAAAAAAAAACAAAAAAD9//v//////wAA///+//3/AAD5//n//v/0//n/+/////7///8BAAAAAQABAAAA/v8BAAEAAQAAAAAAAQAAAAEA/////wEAAAACAAUABAAEAAAA/v8AAAEAAgABAAEAAwAEAAIA///+//7/AAACAAQABAACAAAAAAAAAAAAAAAAAP///v8AAAAAAAD///3//f/+/wAA/f/+////AAAAAAAAAAACAAMABQAEAAMAAgABAAEAAQADAAIAAQAAAAAAAAD+//3/AAACAAQABQD//wMAAwAAAAIAAAAAAP7//f/9//7///////7//f/9//z/+//8//3//f///wAA///9//r/+//8//7//f/+//3//f/9//3//f/9//z/+//9//v/+//7//r/+//7//3//v///////f/9//z//f/8//3//v///////f/9//v//f/7//z//v//////AAABAP//AAAAAAAAAgAAAAIAAgABAAEAAwAEAAIAAgADAAUABQAEAAMAAAD+////+//7//j/9//8////AAAAAAAA/v/8//7//f/+////AAAAAP7//f/8//r/+v/7//v//P/9////AAD/////AAAAAP7/AAAAAP///v////3//f/+//3//f/+//////8AAP///v/9//3//P/8//v/+//7//v////+////AAD//////v/7//z//P/8/wAAAgADAAEA///+/////v///wAA/v8AAAAAAAD8//v//f8AAAgADAAOAAwADAAJAAUA+f/w/+L/1//m//3/BwAIAAIAAAAAAP//AAD+////AAACAAEAAgABAP//AAAAAAAAAQABAAAA/v/+//3//P/6//z//f/8//7//v/9//3/+//7//z/+//7//z//v/9//7//f/9//z//f///wAA///9//3//v8AAAAAAAD//wAAAAAAAAAA/////////////////f/+//3//v////////8AAAAAAAABAAAAAAD9////AQABAAIAAQAAAAMAAwAAAAAA+f/5//z/AAAAAAAAAAAAAAAAAAAAAP7//f/+///////+/wAAAAAAAAAA/v8AAAAAAAABAAAAAAAAAP7//P/9////AAAAAAAAAQAAAAAA/v/9//3///8AAAAAAAD///7//v/+//7//v/+//z//f/////////9//3//f/+//3//f/+//7///////////8AAAAAAQAAAAAAAAAAAAAAAQACAAEAAAD///7//f/7//3/AAABAAQABwAIAAcABQACAAEAAAD///7//v///wAAAQABAAAAAAAAAAAAAAAAAAEAAAABAAEAAQAAAP////8AAAAAAAAAAP///////wAA//////7//f/+//v//P/9//z//f/+//////8AAP///v/+//7//v/+//////8AAP7//f/8//v//P/7//3//v//////AAD///7////+/wAAAAAAAAAA///+////AAAAAAAAAAACAAQAAwACAAAA/f/7//r/+v/7////AAAAAAIAAAAAAP///P/8/////v8AAAAAAQAAAP7//f/7//r/+f/7//v//////wEAAAAAAAAAAgABAAAAAgAAAAAAAAAAAP//AAAAAAAAAAAAAAAAAAABAAAAAAAAAAAA//8AAP//AAD//wEAAgAAAAEAAQAAAAAAAAD+/wAAAAD//wIAAwAEAAEAAAABAAAA//////3//v/8//3/+v/5//7//v8FAAcABwAFAAQA+//9/+//9P/r/8v/3//0/wkADgALAAUABQAAAP/////9/wEAAgADAAMAAwABAAEAAAAAAAAAAgAAAP//////////+//9/////f/+//////////7//f///////f/+/wAAAAAAAAAAAAAAAAAAAQABAAIAAAAAAAAAAAAAAAEAAAAAAAAAAQABAAAAAAAAAAAA//////7//v///wAAAAABAAEAAAACAAEAAgABAAAAAAAAAAIAAwADAAMAAQABAAIAAAAAAPz/+f/8//7/AAAAAAAAAAD////////9//3//v/////////+/////v////7//P////7///8AAP/////+//z//P/+//7/AAD//wAAAAD///7//f/9//7/AAAAAAAA///+//7//v/+//7//v/9//z//v/9//7//f/8//3//f/+//3//v/+//7/////////AAAAAAEAAQAAAAAAAAABAAEAAwADAAEAAAAAAP///v/9/wAAAQADAAcABgAHAAgAAgABAAAAAAAAAAAAAAABAAMAAwACAAAAAQABAAAAAAACAAEAAgACAAIAAQAAAAAAAAAAAAAAAAAAAAAA//////////////3////+//3//v/9//3//f/+//////////3//v/+//7//v//////AAAAAP7//f/7//z//P/7//3//f/+//7////9//7//v/9/wAA//8AAP7//f/8//3/////////AAABAAIAAAAAAP3/+//6//v//P8AAAIAAQACAAEAAAAAAP7//P//////AAAAAAAAAAD///7//f/8//v//P/8//3//////wAA//8AAAAAAgAAAAIAAgAAAAAAAAAAAAAAAQAAAAEAAQABAAAAAwACAAEAAAAAAAAA/////wAAAAAAAAMAAQAAAAEAAAAAAAAA/f/+/////f///wEAAgACAAAAAAAAAAAAAAAAAP3//////wAA/P////7/AAAFAAUABgAFAAAAAgABAP7//P/S/8//4v/5/wQABAABAAAAAgD//wEA/f8AAAAAAwADAAUAAwACAAEAAQAAAAAAAQD//wAAAAAAAP3//P////////8AAAAA///+//3//f////3//f/+/wAA//8AAP//AAD+////AAAAAAEAAAAAAAAAAgABAAIAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAP7//////wAAAQABAAAAAQABAAEAAgABAAEA//8AAAMAAwADAAEA//8AAAAAAAD9//v//P///wAAAAAAAAAA///+/////v/9//7///8AAAAA/v////////8AAP3///8AAP//AAAAAP/////+//3////////////+/wAA///+//3//P/+////AAAAAAAA///+//3//v/+//7////+//7//////wAA/v/+//7////+//7////+////AAAAAAAAAAABAAIAAQAAAAAAAAABAAEAAgAAAAAAAAD//////f///wEAAQAEAAIAAQAGAAEAAAABAAAA//////////8BAAALine truncated
" Your browser does not support the audio element.\n",
" </audio>\n",
" "
],
"text/plain": [
"<IPython.lib.display.Audio object>"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"vocals = ve.post_process(vocals, sr=ORIGINAL_SR, target_sr=TARGET_SR)\n",
"\n",
"show_audio(vocals, TARGET_SR)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Phoneme Recognition"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Some weights of Wav2Vec2ForCTC were not initialized from the model checkpoint at facebook/wav2vec2-large-960h-lv60-self and are newly initialized: ['wav2vec2.masked_spec_embed']\n",
"You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
]
}
],
"source": [
"from lsync.phoneme_recognizer import PhonemeRecognizer\n",
"\n",
"phone_rec = PhonemeRecognizer(lang=lang)\n",
"emission, pred, transcription, frame_duration = phone_rec.recognize(vocals)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[\"BARR I'M ASLEEP AT THE WHEEL AND THERE'S A CAB IN THE HIWAY YET WIC SAT THROUGH THE NIFE LIKE A BEOT I MA TRAN HOW MUCH FETHER IS IT AGEIS SO EXQUISITE WE CAN'T STOP IT TIM NOW YOU DON'T WANT A CHANCE SO LOOK INRIGHT THOG YOUU GOS THAT YOU ER WIT DOAO NOW TE IS ON FIRE AND THE DESERT IS FREEZING WONDER HOW I'M LIVE WHEN I'M NOT EVEN BREATHING FATA IS IT ACHING SO EXQUISITE WE CAN'T STOP INTIDE NOW YOU DON'TANT TAN LOK A THE CLOTHES THAT YOU WEAR WITH DOWAT O STAR BUT THE PEOPLE BUT SAY WHAT THEY WAN TO THA'S TE TRUTH THAT IT IS WE DO RELYCA I\"]\n"
]
}
],
"source": [
"print(transcription)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Visualization of Phoneme"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['<pad>', '<s>', '</s>', '<unk>', '|', 'E', 'T', 'A', 'O', 'N', 'I', 'H', 'S', 'R', 'D', 'L', 'U', 'M', 'W', 'C', 'F', 'G', 'Y', 'P', 'B', 'V', 'K', \"'\", 'X', 'J', 'Q', 'Z']\n",
"torch.Size([8988, 32])\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAjIAAAD0CAYAAACB4szZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAABDoElEQVR4nO3de3wUVZ4F8NOPdCeQFwRJCBAIiIKigqAIgqIiDOooAwOKjInoqIOAPEQHXEFREVFnFZWHOg4MgyyKioCOsiygoosPUBkFiSg4oJIgYhIg5NHdtX+4NHTqhFy6A51OzvfzyedDKvW4dW9V5dI5/WuHZVkWRERERGKQM9oNEBEREQmXJjIiIiISszSRERERkZiliYyIiIjELE1kREREJGZpIiMiIiIxSxMZERERiVmayIiIiEjM0kRGREREYpYmMiJi891338HhcGD+/PnRbkqId955Bw6HA6+88kq0m3JSzJ8/Hw6HAxs2bKixffbu3RsdO3asdj12Ddx///1wOBwh67Vu3Ro33nhjjbVP5HhpIiP11uFfEuxr4sSJ0W6eSEzasmUL7r//fnz33XfRborUE+5oN0Ak2h544AFkZ2eHLDP5H2td1qpVKxw6dAhxcXHRbopEiek1kJeXB6fzyP+Jt2zZgqlTp6J3795o3br1CW6liCYyIujfvz+6du1qtG5paSk8Hk/Ig7sucjgciI+Pj3YzoubgwYNo2LBhje83EAigvLw8JvrW9Brwer0noTUiVavbT2ORCBzOYyxevBj33nsvmjdvjgYNGqC4uBj79u3DhAkTcNZZZyExMRHJycno378/Nm3aRPfx8ssvY+rUqWjevDmSkpLw+9//HkVFRSgrK8PYsWPRtGlTJCYmYvjw4SgrK7O1ZeHChejSpQsSEhLQuHFjXHfdddi1a1e15zB+/HikpaXh6A+5Hz16NBwOB5566qngsoKCAjgcDsyZMwcAz0fk5+dj+PDhaNGiBbxeL5o1a4ZrrrnG9ieEt956C7169ULDhg2RlJSEK6+8Eps3bzbpchQWFmLcuHFo3bo1vF4vWrRogZycHOzduzdkvUAggGnTpqFFixaIj4/HZZddhm+++SZknXXr1mHw4MHIysqC1+tFy5YtMW7cOBw6dChkvRtvvBGJiYn49ttvccUVVyApKQnDhg2rso2HcyJbt27FkCFDkJycjLS0NIwZMwalpaUh6zocDowaNQovvvgizjzzTHi9Xrz99tsAgM8++wz9+/dHcnIyEhMTcdlll+HDDz+kxywpKcFtt92GtLQ0JCcnIycnB7/88kvIOsuWLcOVV16JzMxMeL1etG3bFg8++CD8fj/d58aNG9GjRw8kJCQgOzsbc+fODfm5aU7q6IzM/PnzMXjwYADAJZdcEvxT7TvvvIPc3Fw0adIEFRUVtn307dsXp59++jGPI1IVvSIj9V5RUZHtF2WTJk2C/37wwQfh8XgwYcIElJWVwePxYMuWLXj99dcxePBgZGdno6CgAM8++ywuvvhibNmyBZmZmSH7mz59OhISEjBx4kR88803ePrppxEXFwen04lffvkF999/Pz788EPMnz8f2dnZmDJlSnDbadOmYfLkyRgyZAj++Mc/4qeffsLTTz+Niy66CJ999hlSU1OrPLdevXrhiSeewObNm4N/Llu3bh2cTifWrVuHO+64I7gMAC666KIq9zVo0CBs3rwZo0ePRuvWrbFnzx6sWrUKO3fuDP4J4R//+Adyc3PRr18/zJgxAyUlJZgzZw569uyJzz777Jh/ajhw4AB69eqFr776CjfddBPOPfdc7N27F8uXL8f3338fMiaPPPIInE4nJkyYgKKiIjz66KMYNmwYPvroo+A6S5YsQUlJCUaMGIG0tDR8/PHHePrpp/H9999jyZIlIcf2+Xzo168fevbsiccffxwNGjSosp2HDRkyBK1bt8b06dPx4Ycf4qmnnsIvv/yCBQsWhKy3Zs0avPzyyxg1ahSaNGmC1q1bY/PmzejVqxeSk5Nx9913Iy4uDs8++yx69+6Nd999F926dQvZx6hRo5Camor7778feXl5mDNnDv79738HJ8rAr5OIxMREjB8/HomJiVizZg2mTJmC4uJiPPbYYyH7++WXX3DFFVdgyJAhGDp0KF5++WWMGDECHo8HN910U7XnXpWLLroId9xxB5566incc8896NChAwCgQ4cOuOGGG7BgwQKsXLkSV111VXCb/Px8rFmzBvfdd1/Yx5V6zhKpp+bNm2cBoF+WZVlr1661AFht2rSxSkpKQrYtLS21/H5/yLIdO3ZYXq/XeuCBB4LLDu+jY8eOVnl5eXD50KFDLYfDYfXv3z9kH927d7datWoV/P67776zXC6XNW3atJD1vvjiC8vtdtuWV7Znzx4LgDV79mzLsiyrsLDQcjqd1uDBg6309PTgenfccYfVuHFjKxAIBM8FgDVv3jzLsizrl19+sQBYjz32WJXH2r9/v5WammrdcsstIcvz8/OtlJQU2/LKpkyZYgGwXnvtNdvPDrfrcH926NDBKisrC/585syZFgDriy++CC6rPGaWZVnTp0+3HA6H9e9//zu4LDc31wJgTZw48ZjtO+y+++6zAFhXX311yPLbb7/dAmBt2rQpuAyA5XQ6rc2bN4esO2DAAMvj8VjffvttcNmPP/5oJSUlWRdddFFw2eFrtEuXLiHXz6OPPmoBsJYtW3bM873tttusBg0aWKWlpcFlF198sQXA+stf/hJcVlZWZnXq1Mlq2rRp8DiVr4Gjz/1orVq1snJzc4PfL1myxAJgrV27NmQ9v99vtWjRwrr22mtDlv/nf/6n5XA4rO3bt9vaL2JCf1qSem/WrFlYtWpVyNfRcnNzkZCQELLM6/UGczJ+vx8///wzEhMTcfrpp+PTTz+1HSMnJyckNNmtWzdYlmX732+3bt2wa9cu+Hw+AMBrr72GQCCAIUOGYO/evcGvjIwMtGvXDmvXrj3muZ1yyilo37493nvvPQDABx98AJfLhbvuugsFBQXYtm0bgF9fkenZs6ftrbWHJSQkwOPx4J133rH9SeOwVatWobCwEEOHDg1pq8vlQrdu3apt66uvvopzzjkHv/vd72w/q9yu4cOHw+PxBL/v1asXAGD79u0hbT7s4MGD2Lt3L3r06AHLsvDZZ5/ZjjFixIhjtq+ykSNHhnw/evRoAMA///nPkOUXX3wxzjjjjOD3fr8f//3f/40BAwagTZs2weXNmjXD9ddfj/fffx/FxcUh+7j11ltDrp8RI0bA7XaHHOvo892/fz/27t2LXr16oaSkBFu3bg3Zn9vtxm233Rb83uPx4LbbbsOePXuwceNG4z44Hk6nE8OGDcPy5cuxf//+4PIXX3wRPXr0sAXuRUzpT0tS751//vnHDPuyB2wgEMDMmTMxe/Zs7NixIySHkJaWZls/Kysr5PuUlBQAQMuWLW3LA4EAioqKkJaWhm3btsGyLLRr14627fAvtwMHDuDAgQPB5S6XC6eccgqAX3/JH/6Ft27dOnTt2hVdu3ZF48aNsW7dOqSnp2PTpk24/vrrq+wDr9eLGTNm4M4770R6ejouuOACXHXVVcjJyUFGRgYABCdFl156Kd1HcnJylfsHgG+//RaDBg065jqHVe7PRo0aAUDIJGvnzp2YMmUKli9fbpt8FRUVhXzvdrvRokULo2MfVnlM2rZtC6fTacsMVb5+fvrpJ5SUlNBMSIcOHRAIBLBr1y6ceeaZVR4rMTERzZo1CznW5s2bce+992LNmjW2iVDl883MzLSFmU877TQAv2ZjLrjgAnLGkcvJycGMGTOwdOlS5OTkIC8vDxs3brTlc0SOhyYyItWo/GoMADz88MOYPHkybrrpJjz44INo3LgxnE4nxo4di0AgYFvf5XLRfVe13Pr/cG4gEIDD4cBbb71F101MTAQAPP7445g6dWpweatWrYK/5Hr27Innn38e27dvx7p169CrVy84HA707NkT69atQ2ZmJgKBQPBVjaqMHTsWv/3tb/H6669j5cqVmDx5MqZPn441a9agc+fOwfP+xz/+EZzcHM3trrnHTXX95vf7cfnll2Pfvn3485//jPbt26Nhw4b44YcfcOONN9rG6OhX2MJ1rFezTrTCwkJcfPHFSE5OxgMPPIC2bdsiPj4en376Kf785z/TazIazjjjDHTp0gULFy5ETk4OFi5cCI/HgyFDhkS7aRLDNJERCcMrr7yCSy65BC+88ELI8sLCwpBQaqTatm0Ly7KQnZ0d/B8zk5OTg549ewa/P/qX5+EJyqpVq/DJJ58Ei/1ddNFFmDNnTvB/5126dDFqz5133ok777wT27ZtQ6dOnfCXv/wFCxcuRNu2bQEATZs2RZ8+fcI61y+//PK4t2O++OILfP311/j73/+OnJyc4PLKfzaMxLZt20Jebfnmm28QCASqrZ1yyimnoEGDBsjLy7P9bOvWrXA6nbZX6rZt24ZLLrkk+P2BAwewe/duXHHFFQB+fXfczz//jNdeey0ksL1jxw7ahh9//NH2FvOvv/4aACKu/VLVhO6wnJwcjB8/Hrt378aiRYtw5ZVXBl9REwmHMjIiYXC5XCFvaQZ+fZfMDz/8UKPHGThwIFwuF6ZOnWo7nmVZ+PnnnwEAbdq0QZ8+fYJfF154YXC97OxsNG/eHE888QQqKiqCP+vVqxe+/fZbvPLKK7jggguO+YpJSUmJ7a3Fbdu2RVJSUvDt4v369UNycjIefvhh+hbbn3766ZjnOmjQIGzatAlLly61/azyuVfn8Cs2R29nWRZmzpx5XPs5llmzZoV8//TTTwP4tS5RdW3r27cvli1bFvKnoYKCAixatAg9e/a0/RnuueeeC+nTOXPmwOfzBY/Fzre8vByzZ8+mbfD5fHj22WdD1n322WdxyimnGE1oj+Xw5KiwsJD+fOjQoXA4HBgzZgy2b9+OP/zhDxEdT0SvyIiE4aqLine truncated
"text/plain": [
"<Figure size 640x480 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import matplotlib.pyplot as plt\n",
"labels = phone_rec.get_labels()\n",
"\n",
"print(labels)\n",
"print(emission.shape)\n",
"\n",
"plt.imshow(emission[0:100].T)\n",
"plt.title(\"Frame-wise char probability\")\n",
"plt.xlabel(\"Time\")\n",
"plt.ylabel(\"Labels\")\n",
"# plt.yticks(range(len(labels)), labels)\n",
"plt.show()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Alignment"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Generate trellis and find optimal path via backtracking"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[10, 27, 17, 4, 7, 12, 15, 5, 5, 23, 4, 7, 6, 4, 6, 11, 5, 4, 18, 11, 5, 5, 15, 4, 7, 9, 14, 4, 6, 11, 5, 13, 5, 27, 12, 4, 7, 4, 19, 16, 13, 25, 5, 4, 10, 9, 4, 6, 11, 5, 4, 11, 10, 21, 11, 18, 7, 22, 4, 22, 5, 6, 4, 18, 5, 4, 6, 5, 7, 13, 4, 6, 11, 13, 8, 16, 21, 11, 4, 6, 11, 5, 4, 9, 10, 21, 11, 6, 4, 15, 10, 26, 5, 4, 7, 4, 23, 5, 22, 8, 6, 5, 4, 17, 10, 21, 13, 7, 10, 9, 5, 4, 11, 8, 18, 4, 17, 16, 19, 11, 4, 20, 16, 13, 6, 11, 5, 13, 4, 10, 12, 4, 10, 6, 4, 4, 7, 19, 11, 10, 9, 21, 4, 4, 12, 8, 4, 5, 28, 30, 16, 10, 12, 10, 6, 5, 4, 10, 4, 11, 8, 23, 5, 4, 18, 5, 4, 19, 7, 9, 4, 12, 6, 8, 23, 4, 10, 9, 4, 6, 10, 17, 5, 4, 4, 9, 8, 18, 4, 22, 8, 16, 4, 14, 8, 9, 27, 6, 4, 18, 7, 9, 6, 4, 6, 8, 4, 14, 7, 9, 19, 5, 4, 4, 27, 19, 7, 16, 12, 5, 4, 18, 5, 27, 13, 5, 4, 15, 8, 8, 26, 10, 9, 21, 4, 13, 10, 21, 11, 6, 4, 6, 11, 13, 8, 16, 21, 11, 4, 22, 8, 16, 4, 8, 11, 4, 8, 11, 4, 4, 7, 9, 14, 4, 6, 11, 5, 4, 19, 15, 8, 6, 11, 5, 12, 4, 6, 11, 7, 6, 4, 22, 8, 16, 4, 18, 5, 7, 13, 4, 4, 18, 5, 4, 14, 8, 9, 27, 6, 4, 17, 5, 7, 9, 4, 6, 8, 4, 12, 6, 7, 13, 5, 4, 4, 9, 8, 18, 4, 6, 11, 5, 4, 12, 26, 22, 4, 10, 12, 4, 8, 9, 4, 20, 10, 13, 5, 4, 7, 9, 14, 4, 6, 11, 5, 4, 14, 5, 12, 5, 13, 6, 4, 10, 12, 4, 20, 13, 5, 5, 31, 10, 9, 21, 4, 18, 8, 9, 14, 5, 13, 4, 11, 8, 18, 4, 10, 27, 17, 4, 7, 15, 10, 25, 5, 4, 4, 11, 5, 9, 4, 10, 27, 17, 4, 9, 8, 6, 4, 5, 25, 5, 9, 4, 24, 13, 5, 7, 6, 11, 10, 9, 21, 4, 11, 8, 18, 4, 17, 16, 19, 11, 4, 20, 16, 13, 6, 11, 5, 13, 4, 10, 12, 4, 10, 6, 4, 4, 7, 19, 11, 10, 9, 21, 4, 4, 12, 8, 4, 5, 28, 30, 16, 10, 12, 10, 6, 5, 4, 10, 4, 11, 8, 23, 5, 4, 18, 5, 4, 19, 7, 9, 4, 12, 6, 8, 23, 4, 10, 9, 4, 6, 10, 17, 5, 4, 4, 9, 8, 18, 4, 22, 8, 16, 4, 14, 8, 9, 27, 6, 4, 18, 7, 9, 6, 4, 6, 8, 4, 14, 7, 9, 19, 5, 4, 4, 27, 19, 7, 16, 12, 5, 4, 18, 5, 27, 13, 5, 4, 15, 8, 8, 26, 10, 9, 21, 4, 13, 10, 21, 11, 6, 4, 6, 11, 13, 8, 16, 21, 11, 4, 22, 8, 16, 4, 8, 11, 4, 8, 11, 4, 4, 7, 9, 14, 4, 6, 11, 5, 4, 19, 15, 8, 6, 11, 5, 12, 4, 6, 11, 7, 6, 4, 22, 8, 16, 4, 18, 5, 7, 13, 4, 4, 18, 5, 4, 14, 8, 9, 27, 6, 4, 17, 5, 7, 9, 4, 6, 8, 4, 12, 6, 7, 13, 5, 4, 8, 11, 4, 8, 11, 4, 4, 24, 16, 6, 4, 6, 11, 5, 4, 23, 5, 8, 23, 15, 5, 4, 22, 8, 16, 4, 26, 9, 8, 18, 4, 4, 6, 11, 5, 22, 4, 12, 7, 22, 4, 18, 11, 7, 6, 4, 6, 11, 5, 22, 4, 18, 7, 9, 6, 4, 6, 8, 4, 8, 11, 4, 8, 11, 4, 4, 27, 19, 7, 16, 12, 5, 4, 6, 11, 5, 4, 6, 13, 16, 6, 11, 4, 8, 20, 4, 10, 6, 4, 10, 12, 4, 4, 18, 5, 4, 14, 8, 9, 27, 6, 4, 13, 5, 7, 15, 15, 22, 4, 19, 7, 13, 5, 4]\n",
"[(10, 'I'), (27, \"'\"), (17, 'M'), (4, '|'), (7, 'A'), (12, 'S'), (15, 'L'), (5, 'E'), (5, 'E'), (23, 'P'), (4, '|'), (7, 'A'), (6, 'T'), (4, '|'), (6, 'T'), (11, 'H'), (5, 'E'), (4, '|'), (18, 'W'), (11, 'H'), (5, 'E'), (5, 'E'), (15, 'L'), (4, '|'), (7, 'A'), (9, 'N'), (14, 'D'), (4, '|'), (6, 'T'), (11, 'H'), (5, 'E'), (13, 'R'), (5, 'E'), (27, \"'\"), (12, 'S'), (4, '|'), (7, 'A'), (4, '|'), (19, 'C'), (16, 'U'), (13, 'R'), (25, 'V'), (5, 'E'), (4, '|'), (10, 'I'), (9, 'N'), (4, '|'), (6, 'T'), (11, 'H'), (5, 'E'), (4, '|'), (11, 'H'), (10, 'I'), (21, 'G'), (11, 'H'), (18, 'W'), (7, 'A'), (22, 'Y'), (4, '|'), (22, 'Y'), (5, 'E'), (6, 'T'), (4, '|'), (18, 'W'), (5, 'E'), (4, '|'), (6, 'T'), (5, 'E'), (7, 'A'), (13, 'R'), (4, '|'), (6, 'T'), (11, 'H'), (13, 'R'), (8, 'O'), (16, 'U'), (21, 'G'), (11, 'H'), (4, '|'), (6, 'T'), (11, 'H'), (5, 'E'), (4, '|'), (9, 'N'), (10, 'I'), (21, 'G'), (11, 'H'), (6, 'T'), (4, '|'), (15, 'L'), (10, 'I'), (26, 'K'), (5, 'E'), (4, '|'), (7, 'A'), (4, '|'), (23, 'P'), (5, 'E'), (22, 'Y'), (8, 'O'), (6, 'T'), (5, 'E'), (4, '|'), (17, 'M'), (10, 'I'), (21, 'G'), (13, 'R'), (7, 'A'), (10, 'I'), (9, 'N'), (5, 'E'), (4, '|'), (11, 'H'), (8, 'O'), (18, 'W'), (4, '|'), (17, 'M'), (16, 'U'), (19, 'C'), (11, 'H'), (4, '|'), (20, 'F'), (16, 'U'), (13, 'R'), (6, 'T'), (11, 'H'), (5, 'E'), (13, 'R'), (4, '|'), (10, 'I'), (12, 'S'), (4, '|'), (10, 'I'), (6, 'T'), (4, '|'), (4, '|'), (7, 'A'), (19, 'C'), (11, 'H'), (10, 'I'), (9, 'N'), (21, 'G'), (4, '|'), (4, '|'), (12, 'S'), (8, 'O'), (4, '|'), (5, 'E'), (28, 'X'), (30, 'Q'), (16, 'U'), (10, 'I'), (12, 'S'), (10, 'I'), (6, 'T'), (5, 'E'), (4, '|'), (10, 'I'), (4, '|'), (11, 'H'), (8, 'O'), (23, 'P'), (5, 'E'), (4, '|'), (18, 'W'), (5, 'E'), (4, '|'), (19, 'C'), (7, 'A'), (9, 'N'), (4, '|'), (12, 'S'), (6, 'T'), (8, 'O'), (23, 'P'), (4, '|'), (10, 'I'), (9, 'N'), (4, '|'), (6, 'T'), (10, 'I'), (17, 'M'), (5, 'E'), (4, '|'), (4, '|'), (9, 'N'), (8, 'O'), (18, 'W'), (4, '|'), (22, 'Y'), (8, 'O'), (16, 'U'), (4, '|'), (14, 'D'), (8, 'O'), (9, 'N'), (27, \"'\"), (6, 'T'), (4, '|'), (18, 'W'), (7, 'A'), (9, 'N'), (6, 'T'), (4, '|'), (6, 'T'), (8, 'O'), (4, '|'), (14, 'D'), (7, 'A'), (9, 'N'), (19, 'C'), (5, 'E'), (4, '|'), (4, '|'), (27, \"'\"), (19, 'C'), (7, 'A'), (16, 'U'), (12, 'S'), (5, 'E'), (4, '|'), (18, 'W'), (5, 'E'), (27, \"'\"), (13, 'R'), (5, 'E'), (4, '|'), (15, 'L'), (8, 'O'), (8, 'O'), (26, 'K'), (10, 'I'), (9, 'N'), (21, 'G'), (4, '|'), (13, 'R'), (10, 'I'), (21, 'G'), (11, 'H'), (6, 'T'), (4, '|'), (6, 'T'), (11, 'H'), (13, 'R'), (8, 'O'), (16, 'U'), (21, 'G'), (11, 'H'), (4, '|'), (22, 'Y'), (8, 'O'), (16, 'U'), (4, '|'), (8, 'O'), (11, 'H'), (4, '|'), (8, 'O'), (11, 'H'), (4, '|'), (4, '|'), (7, 'A'), (9, 'N'), (14, 'D'), (4, '|'), (6, 'T'), (11, 'H'), (5, 'E'), (4, '|'), (19, 'C'), (15, 'L'), (8, 'O'), (6, 'T'), (11, 'H'), (5, 'E'), (12, 'S'), (4, '|'), (6, 'T'), (11, 'H'), (7, 'A'), (6, 'T'), (4, '|'), (22, 'Y'), (8, 'O'), (16, 'U'), (4, '|'), (18, 'W'), (5, 'E'), (7, 'A'), (13, 'R'), (4, '|'), (4, '|'), (18, 'W'), (5, 'E'), (4, '|'), (14, 'D'), (8, 'O'), (9, 'N'), (27, \"'\"), (6, 'T'), (4, '|'), (17, 'M'), (5, 'E'), (7, 'A'), (9, 'N'), (4, '|'), (6, 'T'), (8, 'O'), (4, '|'), (12, 'S'), (6, 'T'), (7, 'A'), (13, 'R'), (5, 'E'), (4, '|'), (4, '|'), (9, 'N'), (8, 'O'), (18, 'W'), (4, '|'), (6, 'T'), (11, 'H'), (5, 'E'), (4, '|'), (12, 'S'), (26, 'K'), (22, 'Y'), (4, '|'), (10, 'I'), (12, 'S'), (4, '|'), (8, 'O'), (9, 'N'), (4, '|'), (20, 'F'), (10, 'I'), (13, 'R'), (5, 'E'), (4, '|'), (7, 'A'), (9, 'N'), (14, 'D'), (4, '|'), (6, 'T'), (11, 'H'), (5, 'E'), (4, '|'), (14, 'D'), (5, 'E'), (12, 'S'), (5, 'E'), (13, 'R'), (6, 'T'), (4, '|'), (10, 'I'), (12, 'S'), (4, '|'), (20, 'F'), (13, 'R'), (5, 'E'), (5, 'E'), (31, 'Z'), (10, 'I'), (9, 'N'), (21, 'G'), (4, '|'), (18, 'W'), (8, 'O'), (9, 'N'), (14, 'D'), (5, 'E'), (13, 'R'), (4, '|'), (11, 'H'), (8, 'O'), (18, 'W'), (4, '|'), (10, 'I'), (27, \"'\"), (17, 'M'), (4, '|'), (7, 'A'), (15, 'L'), (10, 'I'), (25, 'V'), (5, 'E'), (4, '|'), (4, '|'), (11, 'H'), (5, 'E'), (9, 'N'), (4, '|'), (10, 'I'), (27, \"'\"), (17, 'M'), (4, '|'), (9, 'N'), (8, 'O'), (6, 'T'), (4, '|'), (5, 'E'), (25, 'V'), (5, 'E'), (9, 'N'), (4, '|'), (24, 'B'), (13, 'R'), (5, 'E'), (7, 'A'), (6, 'T'), (11, 'H'), (10, 'I'), (9, 'N'), (21, 'G'), (4, '|'), (11, 'H'), (8, 'O'), (18, 'W'), (4, '|'), (17, 'M'), (16, 'U'), (19, 'C'), (11, 'H'), (4, '|'), (20, 'F'), (16, 'U'), (13, 'R'), (6, 'T'), (11, 'H'), (5, 'E'), (13, 'R'), (4, '|'), (10, 'I'), (12, 'S'), (4, '|'), (10, 'I'), (6, 'T'), (4, '|'), (4, '|'), (7, 'A'), (19, 'C'), (11, 'H'), (10, 'I'), (9, 'N'), (21, 'G'), (4, '|'), (4, '|'), (12, 'S'), (8, 'O'), (4, '|'), (5, 'E'), (28, 'X'), (30, 'Q'), (16, 'U'), (10, 'I'), (12, 'S'), (10, 'I'), (6, 'T'), (5, 'E'), (4, '|'), (10, 'I'), (4, '|'), (11, 'H'), (8, 'O'), (23, 'P'), (5, 'E'), (4, '|'), (18, 'W'), (5, 'E'), (4, '|'), (19, 'C'), (7, 'A'), (9, 'N'), (4, '|'), (12, 'S'), (6, 'T'), (8, 'O'), (23, 'P'Line truncated
]
}
],
"source": [
"from lsync.alignment import Aligner\n",
"\n",
"# Preprocessing & tokenization\n",
"tokens = phone_rec.tokenize(lyrics)\n",
"print(tokens)\n",
"\n",
"print([(c, labels[c]) for c in tokens])"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Get trellis as the probability of a lyrics token occur at each time frame\n",
"\n",
"- Dimension: (num_frames + 1, num_tokens + 1)\n",
"- Extra token for start-of-sentence\n",
"- Extra frame for computation simplicity"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"torch.Size([8989, 731])\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAigAAACcCAYAAABVyVDPAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAABQSklEQVR4nO19e5RU1Z3ud07R3eKjGxHoBgXE6KAEfKFiazQzl740xswk0cxohhjiePWGgFExRjEPo6NidKLGd3Jnjc66K8qMMxpzNTrDoGg0HR+MRvFB1JBgIg1GQwMqLXTt+0f1ObUfv73PPlXnVV37W6tXV+3nb+86Vfs7v9fxGGMMDg4ODg4ODg4Fgp+3AA4ODg4ODg4OMhxBcXBwcHBwcCgcHEFxcHBwcHBwKBwcQXFwcHBwcHAoHBxBcXBwcHBwcCgcHEFxcHBwcHBwKBwcQXFwcHBwcHAoHEblLUAtKJfLePvtt7HXXnvB87y8xXFwcHBwcHCwAGMM27Ztw6RJk+D7Zh1JQxKUt99+G5MnT85bDAcHBwcHB4ca8NZbb2G//fYztmlIgrLXXnsBqCywvb09Z2kcasHnOs+uvAgY9EjVhBnuEELtn8+t3ZP2w/OqZb5X6RPUlfzKa3+4jKtj0ntyf/2U93x4fuZ5FWOy/B4AvKpsQkprn19HdTzGtYdfrWfhOjHcL7VViSD2UD+3J9ZL/+U1CGPqLiO+r27t0nutfBHfQbIfsRYm1wljDO+BP9xW2QPN+FnBk2QI3sufEaDKFqxd6sO3C8qVayCmjOFYGlmofRTm1IxRbcNI+bUy2Hz+w+/LO3Zgw1V/H57jJjQkQQl+2Nvb2x1BaUD07vVljGrdI3zvBYdss4EjHpX3nlLuyQTD96vEw/dCAsRKHLHhiYk8NveaRRxIdYMiJQKJgvC5Kwc03y54L5MS/ofSl9qnDWL/rA5x0GSE+Wo7oa08njwneeh4dFvDeBT0pEasZ75XKSLJCcI6nnCFh7lmjuw+T2I+/jqTiQbVV/oM+c+GBZe+BcGwgXFfCKJFEqOQTLHqmBHkiR8juLx4UiJfrrKc1Z+06EU3JEFxaGx4pdLwXb90QDcTCDJSKdYQEqEsICg+mKw94QkJRUR8qL8gaYHUnlTkFzQjXPuKvJKMvnigM9+r/ujxrz0vm8PM8pDXEQuKiFB35rZkQt4v6u48imDYgjpUhc/GSJYI+ahDUzdfmtCRx6g18f1NbT3xc697XSYSoPv8FRlpUhJJnDxJ2ymTLo8p7QX49o//cwTFITPM3+ecyouWUdoDuplAkhH+vUw4gv+eVyEmPHnhiEr4Axv+IFb7pq41CcDNXZGVIyWUZoTrp5pruMPXR5WUCIQGQh+5X+IwDK3ThAB6E4FW1S7MqRZqyYiJnNSxLaTmRz7cDOREPMgM7YK58vhZ0K1DQ6RMpIQfy0jWapBRV6bVkgCAxwxaFHp8hU6QnyVT59IgzpodQXHIBPMnfGXYZ8IXTRdAdnf0RQJFRsI6kYzw9YzTnmhJibS3ghZDnitFKNoSwoek0o4nIIYDnCMm6l2f+TDT+nDUAuWAIiY2yWI6qHRaIM14UQSlZpJmOZ9CToI63QFMEhTiM6/3AK8V8jogvlYOctCfJ0lMqLHrIotSgfazYcpnQ5l/wnE14ynzUqTEtJ6gLsZ30REUh9Qxf+JioFQSNQb83X+AZiMqBv+QUPMg1wVaFaNvh0RWONNKlhCIie+JhMQj2gGinGVGkxKP6OdD++Nocj6tFVH2f+t+hBYojsZDJQzy3hrkNIHoZ9QUDP+PWgf5WejGo8ZIE/JBLclIkRO5P6VNiiQH9UIgT1V9h9Z0w5cR41AEh5qTIqQU5M/Q3sDjCIpDyjhp33Ph+b6oMeBNFwGajZwAIqmgTDn8fxtS4ovjChoMQP8jkuLeB8TEGG3ja7QGo3ytpkS056vrS/wQ4GAmJ+bJtNoTwyFNj8M1iOhrfchHzim108kfQUQUmQyER2mbNigZbeWR6kPSLJObWj8fHXhfkmB8ak95QmLcf6a0IUkztR8UmdHIawNHUBzSxSjuEuMdOoHmJCXDYAYzDoDqXg2X01oRT2PSARfVU60Xf0gy2Hsi4iaUgfIjCWWr/pcP4mp4qse1AbQ/+tT4SSLmuGYTiUo6yL4aGUzkJpbGx9A2Un7twWWQR0N4Mjfv8LJI703aE0VDQBETeT2ypqNeeaNIXvDaRCpMa6U+vyhnWF25M/E45I2TDriw8qLEmXJkPwoezUhWZO2IVKcnIB5Qqvbhf2xYyQ9JiXxICw6oaSP8seOIicm5legDgAjb5MYTflCrAxnNDEmhhnFVk0GVZFH1xv6SLMa+NZAdHUhCJZRZyCOVURoYG1nSgNFnxgSKjAivCS1HrZD3ji/nCYlcxpMKknCI45Nr1vTXacqq43OEzEXxOOQNNqokFphMF80ImbQJ2pPgvy/WGXxK2LDZLDyshTwTwaEuHSgpg9KWkH4knEy8L4n2AA9/9D3lB19Qr6eFGvePXI/psIGmjpLHchyrsTTQEhL+NVVnmEfrn5Hjz4K1pkqu8yAmoOOdU7k2yhi1rpX/zOUyYS8NETam9jrZDOSE2RAYp0FxyAvzD76k8oLKEKvzs0CGd/ZFghwGzL2mfEoq5VD9SjyRmIjmD0/8UcyQEFKkhCQkOlkBoW2lvYaUePz/FNdYMzkR+1MHsfFw1sxrHIfoZ9ybmISCMidYm2c0xCYXs44MHTnUfD6UtsSokZAJZRJyagiJIB/RT+hDkQrD58EoIqMld1xbp0FxyA28SQeAbLrQ5uhoRlBRN1y5rCmhMq3KTqhyO8q0khUUYuLpNSTaAxyAbBLSHZJM52ybBiznociC1V2sYT7tHXxY76nyRRGciDmF9gaSYoqm0o1TBK0JEG0mI+WUiIGqISTa82PUuWbKbKP1Q5EJjE6OsL2BSJjkl7UonlTnCIpDHug99NtAadi0Q6nxTGaNAGmq5osIZU+Gy+Vsq4qPxTAxkZ5ZYzSrZAmOhLASFO0JZa4JYPQD8OTDrUp8Mr37jphLe9h5mnZR5AUG7UccEkTJZ2gn19lqUExQfXGi+2QKw0EvH/KUVkQxW8HQJqyrQY+imS+AQF6kdahjaMw6Np+LjpBQ5h4PzsTjkD16j/gOMEq88owExDnJCmSi8r56gCsJuyQtC+MIjJaUhGMR2pcMIETccAdRVTaVPEU6KerG8wyHbtKIOMQDRGs6qDLN52SY06x9MsugjGGYy3hnbrv/Uh/tgZ0ndPtJaEjEeqmfVEZpO5QxEpBR1yYRMsLPYSIkHtEelZ8xz2lQHLLEvKO/WyUnRht3cBBzZU3oJKsNJ/UIU4wceTN8QAcp3yvjcX2DMfmDnOeNWRIUkkh4CrFQ+3FvPGr9ELUmET+yqR1+EeNGaSlMppgoDYgyj2YPrElS1DyaA5nqH6md8YhD3YRMr9lgTjrhGWnC4vcmeC8QMUZ/riZiaYO42pGoPdckZKPbcn3kOT0mHgPS/J5f1gyqwhEUh5oxb84VlRe+H/3l0vlRBGhmDYqsHZGdXAMEP+xRmhIPxmgYYe6UwXxOTpmsyCDWC1DmIPEOnNZEJLoMRU4bGFPgGw6sKFJBlcceR6qzJQyRhIQgnaZDOXr8ml1Ia0JdWiLlemQqEaeu0RgaBWU+6bW142rcSB25n/C7w2lIfNXZ1uNIjOcB5RgmntgW/z/84Q/44he/iH322QejR4/GrFmz8Nxzz4X1jDF85zvfwcSJEzF69Gj09PTg9ddfF8Z47733sGDBArS3t2PMmDE466yzsH379riiOOSM8ig//GMlH2yU4c/nHhpX8sL3rDTc1/ea7g+lyh8reWCjpPpSkOp+WFMwykN5lLR3Pirf4GFfFOZ7YKNAtgv+yqXKH1+W2l9JnT/4C3/MpfVW+5rG5f4CM9KwNqWyxpw/2+HPNPhsws+IWFMoe6n6X2hf0uwFVy58niVxbHJ+ro1yPXiGP15ebhxFXklGcv0ewHxm/iuxbK5TTt7q98luT4X99StrwvBf2Ee5Rlm1XYlpx4v6o/ZMkC8k8az6F8xLtfP5eia+V/pV3ns+C802Xil4P/zfZ/BLXHlQl1Ym2T/96U84/vjj8Rd/8Rd4+OGHMX78eLz++uvYe++9wzbXXnstbrrpJvzzP/8zpk2bhm9/+9vo7e3FK6+8gt122w0AsGDBAmzcuBErV67Ezp07ceaZZ+Kcc87B3XffHUcchxzRc8JVoonBBjpNQTOC126Qd1Xc3sqmMY22RPFJAarvZZ+WtCHfefqGtRKaEh6KtoTQmshZTIXxsoSNZsCk7YiSP6Is0ofHVkYKnIw6rQjpC2EzfhE+O0oOroz+PERNBOlbQ2odUNF21LpO415qnFMpmcl6zXjBW/4zDjUjldd8XfhS7h/DxOMxxqzpzCWXXIKnnnoKP//5z8l6xhgmTZqECy+8EF//+tcBAAMDA+js7MRdd92F008/Ha+++ipmzJiBZ599FkcddRQA4JFHHsGnPvUp/P73v8ekSZOUcQcHBzE4OBi+37p1KyZPnoyBgQG0t7dbL9YhGcz95NWVFz5db5NrQWnThFyFJCOASEg0B7FASsDd3fFjy6YVCinue3VuiTjpXlPyyG1lchvsC7X+LGBLIjhoyYgFiRD6a9oazTCW14HxeiFJZo2ZUnWy5fB7YJvsLGxPfQ4hSYkgJUmsL8rZ1SYVvYG4eBHjCKSEN/OEdUwcZxjlD3bgN1++2ur8jqVB+elPf4re3l789V//NR5//HHsu++++OpXv4qzzz4bALB+/Xr09/ejp6cn7NPR0YE5c+agr68Pp59+Ovr6+jBmzJiQnABAT08PfN/H008/jc997nPKvMuXL8fll18eR1SHlPA/5i4HRlWvuFhaEOpurgmdZAGRjOjvRj2l3BgZY5kvJCvoonf45GyyTFpfCn4/YoyXJmL5isj9dOs0jGFFuqQ9I8cLine truncated
"text/plain": [
"<Figure size 640x480 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"trellis = Aligner.get_trellis(emission, tokens, blank_id=blank_id)\n",
"print(trellis.shape)\n",
"\n",
"plt.imshow(trellis[1:, 1:][0:3200].T, origin=\"lower\")\n",
"# plt.yticks(range(len(labels)), labels)\n",
"plt.show()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Backtrack to find the most likely alignment path"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[Point(token_index=0, time_index=1139, prob=0.9958117008209229), Point(token_index=0, time_index=1140, prob=0.9963963627815247), Point(token_index=0, time_index=1141, prob=0.9995821118354797), Point(token_index=0, time_index=1142, prob=0.999407172203064), Point(token_index=0, time_index=1143, prob=0.99859219789505), Point(token_index=0, time_index=1144, prob=0.9961361289024353), Point(token_index=0, time_index=1145, prob=0.5588687658309937), Point(token_index=1, time_index=1146, prob=0.979438304901123), Point(token_index=1, time_index=1147, prob=0.9999059438705444), Point(token_index=2, time_index=1148, prob=0.997116208076477), Point(token_index=2, time_index=1149, prob=0.9994359612464905), Point(token_index=2, time_index=1150, prob=0.9999556541442871), Point(token_index=2, time_index=1151, prob=0.3408110737800598), Point(token_index=3, time_index=1152, prob=0.989618718624115), Point(token_index=4, time_index=1153, prob=0.7447339296340942), Point(token_index=4, time_index=1154, prob=0.6670995950698853), Point(token_index=4, time_index=1155, prob=0.9998600482940674), Point(token_index=4, time_index=1156, prob=0.990929901599884), Point(token_index=4, time_index=1157, prob=0.9894214868545532), Point(token_index=4, time_index=1158, prob=0.9987514019012451), Point(token_index=5, time_index=1159, prob=0.9998664855957031), Point(token_index=5, time_index=1160, prob=0.999091386795044), Point(token_index=5, time_index=1161, prob=0.9999831914901733), Point(token_index=5, time_index=1162, prob=0.9751810431480408), Point(token_index=6, time_index=1163, prob=0.997595489025116), Point(token_index=6, time_index=1164, prob=0.9999703168869019), Point(token_index=7, time_index=1165, prob=0.9906569719314575), Point(token_index=7, time_index=1166, prob=0.9999791383743286), Point(token_index=7, time_index=1167, prob=0.999983549118042), Point(token_index=7, time_index=1168, prob=0.9999833106994629), Point(token_index=8, time_index=1169, prob=0.997403085231781), Point(token_index=8, time_index=1170, prob=0.9999816417694092), Point(token_index=9, time_index=1171, prob=0.9987607002258301), Point(token_index=9, time_index=1172, prob=0.9996192455291748), Point(token_index=9, time_index=1173, prob=0.9988454580307007), Point(token_index=9, time_index=1174, prob=0.0005588383646681905), Point(token_index=10, time_index=1175, prob=0.9996165037155151), Point(token_index=10, time_index=1176, prob=0.9995201826095581), Point(token_index=11, time_index=1177, prob=0.9992602467536926), Point(token_index=11, time_index=1178, prob=0.9995985627174377), Point(token_index=11, time_index=1179, prob=0.9999332427978516), Point(token_index=11, time_index=1180, prob=0.9996250867843628), Point(token_index=12, time_index=1181, prob=0.9990606904029846), Point(token_index=12, time_index=1182, prob=0.9997887015342712), Point(token_index=12, time_index=1183, prob=0.9999481439590454), Point(token_index=13, time_index=1184, prob=0.9998838901519775), Point(token_index=13, time_index=1185, prob=0.4883236587047577), Point(token_index=14, time_index=1186, prob=0.9987049102783203), Point(token_index=15, time_index=1187, prob=0.9875154495239258), Point(token_index=15, time_index=1188, prob=0.10120848566293716), Point(token_index=15, time_index=1189, prob=0.9988711476325989), Point(token_index=16, time_index=1190, prob=0.9992558360099792), Point(token_index=16, time_index=1191, prob=0.9998962879180908), Point(token_index=16, time_index=1192, prob=0.9999439716339111), Point(token_index=16, time_index=1193, prob=0.9999592304229736), Point(token_index=16, time_index=1194, prob=0.990139365196228), Point(token_index=17, time_index=1195, prob=0.9992726445198059), Point(token_index=17, time_index=1196, prob=0.4302506744861603), Point(token_index=17, time_index=1197, prob=0.9841821789741516), Point(token_index=18, time_index=1198, prob=0.992885410785675), Point(token_index=19, time_index=1199, prob=0.8441746830940247), Point(token_index=19, time_index=1200, prob=0.4815228581428528), Point(token_index=19, time_index=1201, prob=0.9993516802787781), Point(token_index=20, time_index=1202, prob=0.981346845626831), Point(token_index=20, time_index=1203, prob=0.9998488426208496), Point(token_index=20, time_index=1204, prob=0.9998953342437744), Point(token_index=20, time_index=1205, prob=0.9998507499694824), Point(token_index=20, time_index=1206, prob=0.9998738765716553), Point(token_index=20, time_index=1207, prob=0.9999469518661499), Point(token_index=20, time_index=1208, prob=0.9999761581420898), Point(token_index=20, time_index=1209, prob=0.9999871253967285), Point(token_index=20, time_index=1210, prob=0.999982476234436), Point(token_index=21, time_index=1211, prob=0.9752833247184753), Point(token_index=21, time_index=1212, prob=0.17559798061847687), Point(token_index=21, time_index=1213, prob=0.999976396560669), Point(token_index=21, time_index=1214, prob=0.9999759197235107), Point(token_index=21, time_index=1215, prob=0.9999734163284302), Point(token_index=21, time_index=Line truncated
]
}
],
"source": [
"path = Aligner.backtrack(trellis, emission, tokens)\n",
"print(path)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Convert char path into words with timestamp"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[<00:22.78> I'M, <00:23.05> ASLEEP, <00:23.53> AT, <00:23.71> THE, <00:23.96> WHEEL, <00:24.48> AND, <00:24.62> THERE'S, <00:24.94> A, <00:25.08> CURVE, <00:25.42> IN, <00:25.56> THE, <00:25.82> HIGHWAY, <00:30.10> YET, <00:30.33> WE, <00:30.58> TEAR, <00:30.82> THROUGH, <00:31.10> THE, <00:31.33> NIGHT, <00:31.96> LIKE, <00:32.24> A, <00:32.39> PEYOTE, <00:33.14> MIGRAINE, <00:37.06> HOW, <00:38.00> MUCH, <00:38.85> FURTHER, <00:39.92> IS, <00:40.17> IT, <00:40.89> ACHING, <00:42.61> SO, <00:43.24> EXQUISITE, <00:45.10> I, <00:45.39> HOPE, <00:46.24> WE, <00:46.70> CAN, <00:47.14> STOP, <00:47.53> IN, <00:47.89> TIME, <00:52.30> NOW, <00:52.53> YOU, <00:52.75> DON'T, <00:53.03> WANT, <00:53.25> TO, <00:53.46> DANCE, <00:54.30> 'CAUSE, <00:54.43> WE'RE, <00:54.57> LOOKING, <00:55.02> RIGHT, <00:55.30> THROUGH, <00:55.82> YOU, <00:57.92> OH, <00:58.57> OH, <00:59.10> AND, <00:59.80> THE, <01:00.07> CLOTHES, <01:00.50> THAT, <01:00.84> YOU, <01:01.07> WEAR, <01:01.74> WE, <01:01.98> DON'T, <01:02.34> MEAN, <01:02.74> TO, <01:03.35> STARE, <01:14.42> NOW, <01:14.64> THE, <01:14.90> SKY, <01:15.20> IS, <01:15.40> ON, <01:15.62> FIRE, <01:16.34> AND, <01:16.50> THE, <01:16.70> DESERT, <01:17.26> IS, <01:17.42> FREEZING, <01:21.76> WONDER, <01:22.23> HOW, <01:22.56> I'M, <01:22.82> ALIVE, <01:23.64> HEN, <01:23.85> I'M, <01:24.10> NOT, <01:24.42> EVEN, <01:24.73> BREATHING, <01:28.68> HOW, <01:29.65> MUCH, <01:30.48> FURTHER, <01:31.54> IS, <01:31.79> IT, <01:32.54> ACHING, <01:34.21> SO, <01:34.84> EXQUISITE, <01:36.62> I, <01:37.76> HOPE, <01:37.92> WE, <01:38.35> CAN, <01:38.79> STOP, <01:39.20> IN, <01:39.51> TIME, <01:43.92> NOW, <01:44.18> YOU, <01:44.40> DON'T, <01:44.65> WANT, <01:45.04> TO, <01:45.14> DANCE, <01:45.90> 'CAUSE, <01:46.04> WE'RE, <01:46.20> LOOKING, <01:46.62> RIGHT, <01:46.90> THROUGH, <01:47.29> YOU, <01:49.56> OH, <01:50.71> OH, <01:51.35> AND, <01:51.51> THE, <01:51.71> CLOTHES, <01:52.12> THAT, <01:52.42> YOU, <01:52.68> WEAR, <01:53.32> WE, <01:53.57> DON'T, <01:53.90> MEAN, <01:54.29> TO, <01:54.85> STARE, <01:56.92> OH, <01:58.04> OH, <01:58.65> BUT, <01:58.87> THE, <01:59.10> PEOPLE, <01:59.60> YOU, <01:59.85> KNOW, <02:00.68> THEY, <02:00.92> SAY, <02:01.18> WHAT, <02:01.40> THEY, <02:01.70> WANT, <02:02.15> TO, <02:04.28> OH, <02:05.46> OH, <02:05.98> 'CAUSE, <02:06.29> THE, <02:06.46> TRUTH, <02:06.84> OF, <02:07.07> IT, <02:07.29> IS, <02:08.06> WE, <02:08.30> DON'T, <02:08.68> REALLY, <02:09.59> CARE]\n"
]
}
],
"source": [
"from lsync.util import get_audio_segments, read_text\n",
"from lsync.lyrics_processor import LyricsProcessor\n",
"\n",
"lp = LyricsProcessor(lang=lang)\n",
"processed_lyrics = lp.process(lyrics)\n",
"words = lp.get_words_from_path(processed_lyrics, path, frame_duration)\n",
"\n",
"print(words)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. LRC format generation"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[00:00.00] <00:22.78> I’m <00:23.05> asleep <00:23.53> at <00:23.71> the <00:23.96> wheel <00:24.48> and <00:24.62> there’s <00:24.94> a <00:25.08> curve <00:25.42> in <00:25.56> the <00:25.82> highway\n",
"[00:29.94] <00:30.10> Yet <00:30.33> we <00:30.58> tear <00:30.82> through <00:31.10> the <00:31.33> night <00:31.96> like <00:32.24> a <00:32.39> peyote <00:33.14> migraine\n",
"[00:37.00] <00:37.06> How <00:38.00> much <00:38.85> further <00:39.92> is <00:40.17> it <00:40.89> aching <00:42.61> so <00:43.24> exquisite\n",
"[00:44.92] <00:45.10> I <00:45.39> hope <00:46.24> we <00:46.70> can <00:47.14> stop <00:47.53> in <00:47.89> time\n",
"[00:52.21] <00:52.30> Now <00:52.53> you <00:52.75> don’t <00:53.03> want <00:53.25> to <00:53.46> dance <00:54.30> 'cause <00:54.43> we’re <00:54.57> looking <00:55.02> right <00:55.30> through <00:55.82> you\n",
"[00:57.78] <00:57.92> Oh <00:58.57> oh <00:59.10> and <00:59.80> the <01:00.07> clothes <01:00.50> that <01:00.84> you <01:01.07> wear <01:01.74> we <01:01.98> don’t <01:02.34> mean <01:02.74> to <01:03.35> stare\n",
"[01:14.35] <01:14.42> Now <01:14.64> the <01:14.90> sky <01:15.20> is <01:15.40> on <01:15.62> fire <01:16.34> and <01:16.50> the <01:16.70> desert <01:17.26> is <01:17.42> freezing\n",
"[01:21.70] <01:21.76> Wonder <01:22.23> how <01:22.56> I’m <01:22.82> alive <01:23.64> hen <01:23.85> I’m <01:24.10> not <01:24.42> even <01:24.73> breathing\n",
"[01:28.62] <01:28.68> How <01:29.65> much <01:30.48> further <01:31.54> is <01:31.79> it <01:32.54> aching <01:34.21> so <01:34.84> exquisite\n",
"[01:36.46] <01:36.62> I <01:37.76> hope <01:37.92> we <01:38.35> can <01:38.79> stop <01:39.20> in <01:39.51> time\n",
"[01:43.84] <01:43.92> Now <01:44.18> you <01:44.40> don’t <01:44.65> want <01:45.04> to <01:45.14> dance <01:45.90> 'cause <01:46.04> we’re <01:46.20> looking <01:46.62> right <01:46.90> through <01:47.29> you\n",
"[01:49.42] <01:49.56> Oh <01:50.71> oh <01:51.35> and <01:51.51> the <01:51.71> clothes <01:52.12> that <01:52.42> you <01:52.68> wear <01:53.32> we <01:53.57> don’t <01:53.90> mean <01:54.29> to <01:54.85> stare\n",
"[01:56.76] <01:56.92> Oh <01:58.04> oh <01:58.65> but <01:58.87> the <01:59.10> people <01:59.60> you <01:59.85> know <02:00.68> they <02:00.92> say <02:01.18> what <02:01.40> they <02:01.70> want <02:02.15> to\n",
"[02:04.14] <02:04.28> Oh <02:05.46> oh <02:05.98> 'cause <02:06.29> the <02:06.46> truth <02:06.84> of <02:07.07> it <02:07.29> is <02:08.06> we <02:08.30> don’t <02:08.68> really <02:09.59> care\n"
]
}
],
"source": [
"from lsync.lrc_formatter import LrcFormatter\n",
"\n",
"original_lyrics = read_text(lyrics)\n",
"lrc = LrcFormatter.words2lrc(words, original_lyrics)\n",
"print(lrc)"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [],
"source": [
"from lsync.util import save_lrc, save_words\n",
"\n",
"save_lrc(lrc, audio_name)\n",
"save_words(words, audio_name)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5. Evaluation"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Dataset (Ground Truth)"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [],
"source": [
"ground_truth_path = f\"dataset/jamendolyrics/annotations/words/{audio_name}.csv\""
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"abs err: 0.0881\n",
"percentage of correct segments: 0.9660\n"
]
}
],
"source": [
"from lsync.eval import Eval\n",
"\n",
"abs_err, perc_corr_segs = Eval.evaluate(words, ground_truth_path)\n",
"print(f\"abs err: {abs_err:.4f}\\npercentage of correct segments: {perc_corr_segs:.4f}\")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Batch Evaluation"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>name</th>\n",
" <th>audio</th>\n",
" <th>lyrics</th>\n",
" <th>annotation</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>LUNABLIND_-_Vision__Radio_Edit_</td>\n",
" <td>dataset/jamendolyrics/mp3/LUNABLIND_-_Vision__...</td>\n",
" <td>dataset/jamendolyrics/lyrics/LUNABLIND_-_Visio...</td>\n",
" <td>dataset/jamendolyrics/annotations/words/LUNABL...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>JASON_MILLER_-_CROWD_PLEASER</td>\n",
" <td>dataset/jamendolyrics/mp3/JASON_MILLER_-_CROWD...</td>\n",
" <td>dataset/jamendolyrics/lyrics/JASON_MILLER_-_CR...</td>\n",
" <td>dataset/jamendolyrics/annotations/words/JASON_...</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" name \\\n",
"0 LUNABLIND_-_Vision__Radio_Edit_ \n",
"1 JASON_MILLER_-_CROWD_PLEASER \n",
"\n",
" audio \\\n",
"0 dataset/jamendolyrics/mp3/LUNABLIND_-_Vision__... \n",
"1 dataset/jamendolyrics/mp3/JASON_MILLER_-_CROWD... \n",
"\n",
" lyrics \\\n",
"0 dataset/jamendolyrics/lyrics/LUNABLIND_-_Visio... \n",
"1 dataset/jamendolyrics/lyrics/JASON_MILLER_-_CR... \n",
"\n",
" annotation \n",
"0 dataset/jamendolyrics/annotations/words/LUNABL... \n",
"1 dataset/jamendolyrics/annotations/words/JASON_... "
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from lsync.eval import get_eval_df\n",
"\n",
"eval_df = get_eval_df()\n",
"eval_df.head(2)"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Some weights of Wav2Vec2ForCTC were not initialized from the model checkpoint at facebook/wav2vec2-large-960h-lv60-self and are newly initialized: ['wav2vec2.masked_spec_embed']\n",
"You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
"20it [07:27, 22.36s/it]\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>name</th>\n",
" <th>average_abs_err</th>\n",
" <th>percentage_of_correct_segments</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Lower_Loveday_-_Is_It_Right_</td>\n",
" <td>0.240656</td>\n",
" <td>0.971698</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Ridgway_-_Fire_Inside</td>\n",
" <td>1.025926</td>\n",
" <td>0.842105</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Wordsmith_-_The_Statement</td>\n",
" <td>0.091202</td>\n",
" <td>0.979346</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>The.madpix.project_-_One_Way_Street</td>\n",
" <td>1.905688</td>\n",
" <td>0.896175</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Moon_I_Mean_-_Wrong_Concept</td>\n",
" <td>0.121865</td>\n",
" <td>0.973783</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Kinematic_-_Peyote</td>\n",
" <td>0.086068</td>\n",
" <td>0.972789</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>Quentin_Hannappe_-_Keep_On</td>\n",
" <td>0.114757</td>\n",
" <td>0.982857</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>Explosive_Ear_Candy_-_Like_The_Sun</td>\n",
" <td>0.113410</td>\n",
" <td>0.992453</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>Pure_Mids_-_The_Leader</td>\n",
" <td>3.736285</td>\n",
" <td>0.701754</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>HILA_-_Give_Me_the_Same</td>\n",
" <td>0.213927</td>\n",
" <td>0.919255</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>Cortez_-_Feel__Stripped_</td>\n",
" <td>0.167805</td>\n",
" <td>0.963173</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>Rxbyn_-_Bad_Side</td>\n",
" <td>0.137684</td>\n",
" <td>0.956818</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" name average_abs_err \\\n",
"0 Lower_Loveday_-_Is_It_Right_ 0.240656 \n",
"1 Ridgway_-_Fire_Inside 1.025926 \n",
"2 Wordsmith_-_The_Statement 0.091202 \n",
"3 The.madpix.project_-_One_Way_Street 1.905688 \n",
"4 Moon_I_Mean_-_Wrong_Concept 0.121865 \n",
"5 Kinematic_-_Peyote 0.086068 \n",
"6 Quentin_Hannappe_-_Keep_On 0.114757 \n",
"7 Explosive_Ear_Candy_-_Like_The_Sun 0.113410 \n",
"8 Pure_Mids_-_The_Leader 3.736285 \n",
"9 HILA_-_Give_Me_the_Same 0.213927 \n",
"10 Cortez_-_Feel__Stripped_ 0.167805 \n",
"11 Rxbyn_-_Bad_Side 0.137684 \n",
"\n",
" percentage_of_correct_segments \n",
"0 0.971698 \n",
"1 0.842105 \n",
"2 0.979346 \n",
"3 0.896175 \n",
"4 0.973783 \n",
"5 0.972789 \n",
"6 0.982857 \n",
"7 0.992453 \n",
"8 0.701754 \n",
"9 0.919255 \n",
"10 0.963173 \n",
"11 0.956818 "
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from lsync import LyricsSync\n",
"\n",
"lsync = LyricsSync(lang=lang, blank_id=blank_id)\n",
"Eval.eval_all(eval_df, lsync)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Result of finetuned model in singing domain based on facebook/wav2vec2-base"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Avg abs err: 0.6246593301330764 | Avg percentage: 0.9292648064655545\n"
]
}
],
"source": [
"import pandas as pd\n",
"eval_result = pd.read_csv('eval_finetune.csv')\n",
"print(\"Avg abs err: {} | Avg percentage: {}\".format(\n",
" eval_result.loc[:, 'average_abs_err'].mean(),\n",
" eval_result.loc[:, 'percentage_of_correct_segments'].mean(),\n",
"))"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Result of large model of facebook/wav2vec2-large-960h-lv60-self in speech domain"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Avg abs err: 0.8945763391872722 | Avg percentage: 0.9131257208477407\n"
]
}
],
"source": [
"import pandas as pd\n",
"eval_result = pd.read_csv('eval_large.csv')\n",
"print(\"Avg abs err: {} | Avg percentage: {}\".format(\n",
" eval_result.loc[:, 'average_abs_err'].mean(),\n",
" eval_result.loc[:, 'percentage_of_correct_segments'].mean(),\n",
"))"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "lsync",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.12"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}
+229
View File
@@ -0,0 +1,229 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from datasets import Dataset\n",
"import numpy as np\n",
"import torch\n",
"from lsync.config import TARGET_SR\n",
"import librosa\n",
"from transformers import Wav2Vec2Processor, Wav2Vec2FeatureExtractor, Wav2Vec2CTCTokenizer\n",
"\n",
"import evaluate\n",
"\n",
"metrics = evaluate.load(\"wer\")\n",
"MODEL_ID = \"facebook/wav2vec2-base\"\n",
"\n",
"tokenizer = Wav2Vec2CTCTokenizer(\"vocab.json\", unk_token=\"[UNK]\", pad_token=\"[PAD]\", word_delimiter_token=\"|\")\n",
"feature_extractor = Wav2Vec2FeatureExtractor(feature_size=1, sampling_rate=16000, padding_value=0.0, do_normalize=True, return_attention_mask=False)\n",
"processor = Wav2Vec2Processor(feature_extractor=feature_extractor, tokenizer=tokenizer)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Dataset"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def audio_processing(fp):\n",
" y, sr = librosa.load(fp, sr=TARGET_SR)\n",
" yt, _ = librosa.effects.trim(y, top_db=30)\n",
" return yt\n",
"\n",
"\n",
"def data_preprocess(data):\n",
" audio = audio_processing(data['path'])\n",
" result = {}\n",
" result['input_values'] = processor(\n",
" audio, sampling_rate=TARGET_SR).input_values[0]\n",
" with processor.as_target_processor():\n",
" result[\"labels\"] = processor(data[\"text\"]).input_ids\n",
" return result\n",
"\n",
"\n",
"def prepare_dataset(batch):\n",
" audio = batch[\"path\"]\n",
"\n",
" # batched output is \"un-batched\" to ensure mapping is correct\n",
" batch[\"input_values\"] = processor(\n",
" audio[\"array\"], sampling_rate=audio[\"sampling_rate\"]).input_values[0]\n",
" batch[\"labels\"] = processor(batch[\"text\"]).input_ids\n",
" return batch\n",
"\n",
"def get_datasets():\n",
" dataset_csv_path = \"dataset.csv\"\n",
" dataset = Dataset.from_csv(dataset_csv_path)\n",
" dataset = dataset.filter(lambda x: x[\"text\"] != \"\")\n",
" dataset.cleanup_cache_files()\n",
" dataset = dataset.train_test_split(test_size=0.05, seed=41)\n",
" # smaller_dataset = dataset['test'].train_test_split(test_size=0.05)\n",
" train_dataset = dataset['train']\n",
" test_dataset = dataset['test']\n",
"\n",
" train_dataset = train_dataset.map(\n",
" data_preprocess,\n",
" num_proc=8\n",
" )\n",
" test_dataset = test_dataset.map(\n",
" data_preprocess,\n",
" num_proc=8\n",
" )\n",
" return (train_dataset, test_dataset)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"train_dataset, test_dataset = get_datasets()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Train"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"@dataclass\n",
"class DataCollatorCTCWithPadding:\n",
" processor: Wav2Vec2Processor\n",
" padding: Union[bool, str] = True\n",
" max_length: Optional[int] = None\n",
" max_length_labels: Optional[int] = None\n",
" pad_to_multiple_of: Optional[int] = None\n",
" pad_to_multiple_of_labels: Optional[int] = None\n",
"\n",
" def __call__(self, features: List[Dict[str, Union[List[int], torch.Tensor]]]) -> Dict[str, torch.Tensor]:\n",
" input_features = [{\"input_values\": feature[\"input_values\"]}\n",
" for feature in features]\n",
" label_features = [{\"input_ids\": feature[\"labels\"]}\n",
" for feature in features]\n",
"\n",
" batch = self.processor.pad(\n",
" input_features,\n",
" padding=self.padding,\n",
" max_length=self.max_length,\n",
" pad_to_multiple_of=self.pad_to_multiple_of,\n",
" return_tensors=\"pt\",\n",
" )\n",
" with self.processor.as_target_processor():\n",
" labels_batch = self.processor.pad(\n",
" label_features,\n",
" padding=self.padding,\n",
" max_length=self.max_length_labels,\n",
" pad_to_multiple_of=self.pad_to_multiple_of_labels,\n",
" return_tensors=\"pt\",\n",
" )\n",
"\n",
" labels = labels_batch[\"input_ids\"].masked_fill(\n",
" labels_batch.attention_mask.ne(1), -100)\n",
"\n",
" batch[\"labels\"] = labels\n",
"\n",
" return batch\n",
"\n",
"\n",
"def compute_metrics(pred):\n",
" pred_logits = pred.predictions\n",
" pred_ids = np.argmax(pred_logits, axis=-1)\n",
" pred.label_ids[pred.label_ids == -100] = processor.tokenizer.pad_token_id\n",
" pred_str = processor.batch_decode(pred_ids)\n",
" label_str = processor.batch_decode(pred.label_ids, group_tokens=False)\n",
" wer = metrics.compute(predictions=pred_str, references=label_str)\n",
" return {\"wer\": wer}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Train\n",
"data_collator = DataCollatorCTCWithPadding(\n",
" processor=processor, padding=True)\n",
"\n",
"model = Wav2Vec2ForCTC.from_pretrained(\n",
" MODEL_ID,\n",
" ctc_loss_reduction=\"mean\",\n",
" pad_token_id=processor.tokenizer.pad_token_id,\n",
")\n",
"model.freeze_feature_extractor()\n",
"torch.cuda.empty_cache()\n",
"training_args = TrainingArguments(\n",
" output_dir=\"model\",\n",
" per_device_train_batch_size=2,\n",
" per_device_eval_batch_size=2,\n",
" gradient_accumulation_steps=2,\n",
" num_train_epochs=5,\n",
" fp16=True,\n",
" save_steps=3000,\n",
" eval_steps=20000,\n",
" logging_steps=2000,\n",
" learning_rate=5e-4,\n",
" weight_decay=0.0001,\n",
" warmup_steps=1000,\n",
" save_total_limit=8,\n",
" evaluation_strategy=\"steps\",\n",
" # load_best_model_at_end=True,\n",
" # metric_for_best_model=\"eval_wer\"\n",
")\n",
"\n",
"trainer = Trainer(\n",
" model=model,\n",
" data_collator=data_collator,\n",
" args=training_args,\n",
" train_dataset=train_dataset,\n",
" eval_dataset=test_dataset,\n",
" tokenizer=processor.feature_extractor,\n",
")\n",
"\n",
"trainer.train(resume_from_checkpoint=True)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "lsync",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.12"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}
+1
View File
@@ -0,0 +1 @@
{"'": 1, "a": 2, "b": 3, "c": 4, "d": 5, "e": 6, "f": 7, "g": 8, "h": 9, "i": 10, "j": 11, "k": 12, "l": 13, "m": 14, "n": 15, "o": 16, "p": 17, "q": 18, "r": 19, "s": 20, "t": 21, "u": 22, "v": 23, "w": 24, "x": 25, "y": 26, "z": 27, "|": 0, "[UNK]": 28, "[PAD]": 29}