Commit
First push
This commit is contained in:
commit
c4a020c191
146 files changed
+40796
No files matched your search
Executable
+201
@@ -0,0 +1,201 @@
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright [yyyy] [name of copyright owner]
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
Executable
+47
@@ -0,0 +1,47 @@
|
||||
# Lyrics Sync (lsync)
|
||||
|
||||
This project aims to build a lyrics-to-audio alignment system that can synchronize the audio of a polyphonic song with its lyrics and produce time-aligned lyrics with word-level onset and offset as a `.lrc` file. A deep-learning-based system is developed to approach the problem in three steps, which include separating the vocals, recognizing the singing vocals, and performing forced alignment. For singing vocals recognition, transfer learning is utilized to apply knowledge obtained from the speech domain to the singing domain.
|
||||
|
||||
### Flow
|
||||
|
||||

|
||||
|
||||
|
||||
### Installation
|
||||
|
||||
```shell
|
||||
conda env update -f environment.yml
|
||||
conda activate lsync
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
```python
|
||||
from lsync import LyricsSync
|
||||
|
||||
lsync = LyricsSync()
|
||||
words, lrc = lsync.sync(audio_path, lyrics_path)
|
||||
```
|
||||
|
||||
#### Demo
|
||||
|
||||
Please refer to [demo.ipynb](./demo.ipynb).
|
||||
|
||||
If you want to visualize `.lrc` for evaluation, you can use [Lrc Player](https://github.com/mikezzb/lrc-player).
|
||||
|
||||
### Experiments
|
||||
|
||||
If you want to fine-tune a Wav2Vec2 model for better accuracy in singing domain, please refer to the experiments section below.
|
||||
|
||||
#### Fine-tune Wav2Vec2 model for lyrics transcription
|
||||
|
||||
1. Make a `dataset` folder in root folder
|
||||
2. Download [DALI](https://github.com/gabolsgabs/DALI) dataset and put it it inside `dataset/DALI/v1`
|
||||
* Similarly, you can download jamendolyrics dataset for evaluation and put it in `dataset/jamendolyrics`
|
||||
3. Download all DALI songs using `python get_dataset.py`
|
||||
4. Run `dataset.ipynb` to prepare the DALI for fine-tune tasks
|
||||
* Procedures including vocal extraction, line-level segmentation, and making tokenizer
|
||||
5. Run `train.ipynb` to fine-tune the `facebook/wav2vec2-base` for singing voice recognition
|
||||
6. Run `run.ipynb` to see how to use the `lsync` library for lyrics-to-audio alignment based on the fine-tuned model
|
||||
* Remember to update model path to your model's path inside `lsync/phoneme_recognizer.py`
|
||||
|
||||
Executable
+367
@@ -0,0 +1,367 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Prepare DALI for fine-tune task"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Extract Vocals"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import DALI as dali_code\n",
|
||||
"import os\n",
|
||||
"import torch\n",
|
||||
"from lsync.voice_extractor import VoiceExtractor\n",
|
||||
"from lsync.config import ORIGINAL_SR, TARGET_SR\n",
|
||||
"from lsync.util import save_audio, save_audio_file\n",
|
||||
"from tqdm.notebook import tqdm\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"path_audio = os.path.abspath('dataset/DALI/audio')\n",
|
||||
"path_vocals = os.path.abspath(\"dataset/DALI/processed_audio\")\n",
|
||||
"torch.cuda.is_available()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"converted = set([os.path.splitext(x)[0] for x in os.listdir(path_vocals)])\n",
|
||||
"ve = VoiceExtractor()\n",
|
||||
"\n",
|
||||
"for fname in tqdm(os.listdir(path_audio)):\n",
|
||||
" try:\n",
|
||||
" audio_name, ext = os.path.splitext(fname)\n",
|
||||
" if ext != '.mp3' or audio_name in converted:\n",
|
||||
" continue\n",
|
||||
" file_path = os.path.join(path_audio, fname)\n",
|
||||
" vocals = ve.extract_voice(file_path, post_process=True)\n",
|
||||
" save_audio(vocals, audio_name, sr=TARGET_SR, out_path=path_vocals)\n",
|
||||
" except Exception as e:\n",
|
||||
" print(e)\n",
|
||||
" continue"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Make Speech Recognition Dataset for Fine-tune"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Load DALI annotation"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import DALI as dali_code\n",
|
||||
"import os\n",
|
||||
"from lsync.util import save_audio_file\n",
|
||||
"import pandas as pd\n",
|
||||
"import librosa\n",
|
||||
"\n",
|
||||
"dali_data_path = os.path.abspath('dataset/DALI/v1')\n",
|
||||
"path_audio = os.path.abspath('dataset/DALI/audio')\n",
|
||||
"dali_data = dali_code.get_the_DALI_dataset(dali_data_path, skip=[], keep=[])\n",
|
||||
"dali_info = dali_code.get_info(dali_data_path + '/info/DALI_DATA_INFO.gz')\n",
|
||||
"path_segments = os.path.abspath(\"dataset/DALI/segmented_audio\")\n",
|
||||
"csv_path = os.path.abspath('dali_lines.csv')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Make lyrics line level dataset"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"processed_songs = [os.path.splitext(x)[0] for x in os.listdir(path_vocals)]\n",
|
||||
"\n",
|
||||
"def extract_audio_segment(audio, time_seg, save_path, sr = TARGET_SR):\n",
|
||||
" start_t, end_t = time_seg\n",
|
||||
" start_idx, end_idx = int(start_t * sr), int(end_t * sr)\n",
|
||||
" segment = audio[start_idx:end_idx]\n",
|
||||
" save_audio_file(segment, save_path, sr=sr)\n",
|
||||
" return segment\n",
|
||||
"\n",
|
||||
"def get_segment_fname(id: str, segment_index: int):\n",
|
||||
" return f\"{id}_{segment_index}\"\n",
|
||||
"\n",
|
||||
"def make_dataset():\n",
|
||||
" data_list = []\n",
|
||||
" for id in tqdm(processed_songs):\n",
|
||||
" # Get vocals audio\n",
|
||||
" audio_path = os.path.join(path_vocals, f\"{id}.wav\")\n",
|
||||
" audio, sr = librosa.load(audio_path, sr=TARGET_SR)\n",
|
||||
" # Get annotations\n",
|
||||
" entry = dali_data[id]\n",
|
||||
" anno = entry.annotations['annot']\n",
|
||||
" lines = anno['lines']\n",
|
||||
" # Process each line\n",
|
||||
" for seg_idx, line in enumerate(lines):\n",
|
||||
" # Extract segment\n",
|
||||
" segment_time = line['time']\n",
|
||||
" extract_audio_segment(\n",
|
||||
" audio,\n",
|
||||
" segment_time,\n",
|
||||
" save_path=os.path.join(path_segments, get_segment_fname(id, seg_idx))\n",
|
||||
" )\n",
|
||||
" # Add annotation to dataset\n",
|
||||
" data_list.append((id, seg_idx, line['text']))\n",
|
||||
" \n",
|
||||
" df = pd.DataFrame(data_list, columns=['id', 'segment_index', 'text'])\n",
|
||||
" return df\n",
|
||||
"\n",
|
||||
"df = make_dataset()\n",
|
||||
"df.to_csv(csv_path, index=False)\n",
|
||||
"df.head(3)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Make training dataset & data cleaning"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from num2words import num2words\n",
|
||||
"import re\n",
|
||||
"import numpy as np\n",
|
||||
"\n",
|
||||
"allowed_chars = set('abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ \\'')\n",
|
||||
"\n",
|
||||
"def clean_lyrics(lyrics: str) -> str:\n",
|
||||
" # Find all numbers in the text using regular expression\n",
|
||||
" num_regex = r'\\d+'\n",
|
||||
" numbers = re.findall(num_regex, lyrics)\n",
|
||||
" \n",
|
||||
" # Convert each number to its word form\n",
|
||||
" for num in numbers:\n",
|
||||
" word_form = num2words(int(num))\n",
|
||||
" lyrics = lyrics.replace(num, word_form)\n",
|
||||
"\n",
|
||||
" cleaned = \"\"\n",
|
||||
" for char in lyrics:\n",
|
||||
" if char in allowed_chars:\n",
|
||||
" cleaned += char\n",
|
||||
" elif char == '-':\n",
|
||||
" cleaned += ' '\n",
|
||||
" return cleaned\n",
|
||||
"\n",
|
||||
"dataset_csv_path = \"dataset.csv\"\n",
|
||||
"def transform_dataset(csv_path, path_segments, lang='english'):\n",
|
||||
" df = pd.read_csv(csv_path)\n",
|
||||
" df['path'] = df.agg(lambda x: f\"{path_segments}/{x['id']}_{str(x['segment_index'])}.wav\", axis=1)\n",
|
||||
" delete_idx = []\n",
|
||||
" for idx, row in tqdm(df.iterrows()):\n",
|
||||
" song_id = row['id']\n",
|
||||
" # Get annotations\n",
|
||||
" entry = dali_data[song_id]\n",
|
||||
" meta = entry.info['metadata']\n",
|
||||
" song_lang = meta['language']\n",
|
||||
" # Skip different lang\n",
|
||||
" if song_lang != lang:\n",
|
||||
" delete_idx.append(idx)\n",
|
||||
" continue\n",
|
||||
" # Clean invalid data\n",
|
||||
" if not isinstance(row['text'], str) or len(row['text']) == 0:\n",
|
||||
" delete_idx.append(idx)\n",
|
||||
" continue\n",
|
||||
" # Check for audio validity\n",
|
||||
" y, sr = librosa.load(row['path'], sr=TARGET_SR)\n",
|
||||
" yt, _ = librosa.effects.trim(y, top_db=30)\n",
|
||||
" duration = librosa.get_duration(y=yt, sr=sr)\n",
|
||||
" if duration > 3.1 or duration < 0.15: # CUDA out of memory\n",
|
||||
" delete_idx.append(idx)\n",
|
||||
" continue \n",
|
||||
"\n",
|
||||
" df = df.drop(delete_idx)\n",
|
||||
" df = df.drop(['id', 'segment_index'], axis=1)\n",
|
||||
"\n",
|
||||
" # Text cleaning\n",
|
||||
" df['text'] = df['text'].map(clean_lyrics)\n",
|
||||
" df.replace('', np.nan, inplace=True)\n",
|
||||
" df.dropna(inplace=True)\n",
|
||||
" return df\n",
|
||||
"\n",
|
||||
"df = transform_dataset(csv_path, path_segments)\n",
|
||||
"df.to_csv(dataset_csv_path, index=False)\n",
|
||||
"df.head(10)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Finetune"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Load dataset"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from datasets import load_dataset, Audio, Dataset\n",
|
||||
"\n",
|
||||
"dataset_csv_path = \"dataset.csv\"\n",
|
||||
"dataset = Dataset.from_csv(dataset_csv_path)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Sample of dataset loaded"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import IPython.display as ipd\n",
|
||||
"import numpy as np\n",
|
||||
"import random\n",
|
||||
"import librosa\n",
|
||||
"\n",
|
||||
"rand_int = random.randint(0, len(dataset))\n",
|
||||
"\n",
|
||||
"def audio_processing(fp):\n",
|
||||
" y, sr = librosa.load(fp, sr=TARGET_SR)\n",
|
||||
" return y\n",
|
||||
"\n",
|
||||
"print(dataset[rand_int][\"text\"])\n",
|
||||
"ipd.Audio(data=audio_processing(dataset[rand_int][\"path\"]), autoplay=True, rate=16000)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Data cleaning"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import re\n",
|
||||
"chars_to_ignore_regex = '[\\,\\?\\.\\!\\-\\;\\:\\\"]'\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def remove_special_characters(batch):\n",
|
||||
" batch[\"text\"] = re.sub(chars_to_ignore_regex, '', batch[\"text\"]).lower()\n",
|
||||
" return batch\n",
|
||||
"\n",
|
||||
"dataset = dataset.map(remove_special_characters)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import json\n",
|
||||
"\n",
|
||||
"def make_vocab(dataset):\n",
|
||||
" vocab = set()\n",
|
||||
" for sample in dataset['text']:\n",
|
||||
" for ch in sample:\n",
|
||||
" vocab.add(ch)\n",
|
||||
" vocab_dict = {v: k for k, v in enumerate(sorted(vocab))}\n",
|
||||
" vocab_dict[\"|\"] = vocab_dict[\" \"]\n",
|
||||
" del vocab_dict[\" \"]\n",
|
||||
" vocab_dict[\"[UNK]\"] = len(vocab_dict)\n",
|
||||
" vocab_dict[\"[PAD]\"] = len(vocab_dict)\n",
|
||||
" with open('vocab.json', 'w') as vocab_file:\n",
|
||||
" json.dump(vocab_dict, vocab_file)\n",
|
||||
" return vocab_dict\n",
|
||||
"\n",
|
||||
"vocab_dict = make_vocab(dataset)\n",
|
||||
"with open('vocab.json', 'r') as f:\n",
|
||||
" vocab_dict = json.load(f)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"print(vocab_dict)"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "lsync",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.7.12"
|
||||
},
|
||||
"orig_nbformat": 4
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
Executable
+248
@@ -0,0 +1,248 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Demo"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Using pre-trained model (facebook/wav2vec2-large-960h-lv60-self)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"audio_path = \"dataset/hensen/umbrella/umbrella.wav\"\n",
|
||||
"lyrics_path = \"dataset/hensen/umbrella/umbrella.txt\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Some weights of Wav2Vec2ForCTC were not initialized from the model checkpoint at facebook/wav2vec2-large-960h-lv60-self and are newly initialized: ['wav2vec2.masked_spec_embed']\n",
|
||||
"You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"[00:00.00] <00:27.53> you <00:27.90> have <00:28.21> my <00:28.42> heart\n",
|
||||
"[00:29.40] <00:29.46> and <00:29.62> we'll <00:29.98> never <00:30.32> be <00:30.62> worlds <00:31.12> apart\n",
|
||||
"[00:32.32] <00:32.35> maybe <00:33.07> in <00:33.38> magazines\n",
|
||||
"[00:35.11] <00:35.16> but <00:35.48> you'll <00:35.75> still <00:36.14> be <00:36.39> my <00:36.61> star\n",
|
||||
"[00:37.84] <00:37.88> baby <00:38.38> 'cause <00:38.98> in <00:39.20> the <00:39.42> dark\n",
|
||||
"[00:40.57] <00:40.61> you <00:40.92> can't <00:41.30> see <00:41.61> shiny <00:42.14> cars\n",
|
||||
"[00:43.14] <00:43.17> and <00:43.35> that's <00:43.70> when <00:44.03> you <00:44.38> need <00:44.74> me <00:44.96> there\n",
|
||||
"[00:46.06] <00:46.10> with <00:46.46> you <00:46.88> i'll <00:47.16> always <00:47.82> share\n",
|
||||
"[00:49.14] <00:49.17> because <00:50.56> when <00:50.74> the <00:50.92> sun <00:51.21> shines <00:51.64> we <00:51.93> shine <00:52.32> together\n",
|
||||
"[00:53.14] <00:53.38> told <00:53.74> you <00:54.10> i'll <00:54.42> be <00:54.78> here <00:55.06> forever\n",
|
||||
"[00:56.03] <00:56.10> said <00:56.56> i'll <00:56.92> always <00:57.50> be <00:57.85> your <00:57.98> friend\n",
|
||||
"[00:58.50] <00:58.53> took <00:58.75> an <00:58.96> oath <00:59.20> gonna <00:59.46> stick <00:59.78> it <01:00.02> out <01:00.21> till <01:00.57> the <01:00.84> end\n",
|
||||
"[01:01.56] <01:01.62> now <01:01.92> that <01:02.10> it's <01:02.34> rainin' <01:03.00> more <01:03.30> than <01:03.57> ever\n",
|
||||
"[01:04.32] <01:04.34> know <01:04.71> that <01:05.07> we'll <01:05.37> still <01:05.78> have <01:06.14> each <01:06.34> other\n",
|
||||
"[01:07.09] <01:07.14> you <01:07.29> can <01:07.46> stand <01:07.85> under <01:08.18> my <01:08.62> umbrella\n",
|
||||
"[01:09.84] <01:09.87> you <01:10.01> can <01:10.18> stand <01:10.60> under <01:10.92> my <01:11.35> umbrella\n",
|
||||
"[01:12.62] <01:12.71> ella <01:13.37> ella <01:14.15> ae <01:14.76> ae <01:14.98> ae\n",
|
||||
"[01:16.00] <01:16.06> under <01:16.43> my <01:16.90> umbrella <01:18.20> ella <01:18.87> ella <01:19.64> ae <01:20.32> ae <01:21.00> ae\n",
|
||||
"[01:21.54] <01:21.60> under <01:21.93> my <01:22.40> umbrella <01:23.70> ella <01:24.37> ella <01:25.15> ae <01:25.79> ae <01:26.51> ae\n",
|
||||
"[01:27.04] <01:27.10> under <01:27.48> my <01:27.92> umbrella <01:29.21> ella <01:29.92> ella <01:30.65> ae <01:31.32> ae <01:32.01> ae <01:32.07> ae <01:32.68> ae <01:33.35> ae\n",
|
||||
"[01:33.56] <01:33.60> these <01:34.00> fancy <01:34.50> things\n",
|
||||
"[01:35.65] <01:35.70> will <01:36.06> never <01:36.37> come <01:36.84> in <01:37.06> between\n",
|
||||
"[01:38.37] <01:38.42> you're <01:38.76> part <01:39.01> of <01:39.18> my <01:39.57> entity\n",
|
||||
"[01:41.14] <01:41.18> here <01:41.54> for <01:42.00> infinity\n",
|
||||
"[01:43.90] <01:43.93> when <01:44.12> the <01:44.32> war <01:44.68> has <01:44.98> took <01:45.32> it's <01:45.56> part\n",
|
||||
"[01:46.65] <01:46.68> when <01:46.85> the <01:47.06> world <01:47.37> has <01:47.74> dealt <01:48.07> it's <01:48.34> cards\n",
|
||||
"[01:49.78] <01:49.90> if <01:50.15> the <01:50.48> hand <01:50.87> is <01:51.01> hard\n",
|
||||
"[01:52.35] <01:52.37> together <01:52.90> we'll <01:53.21> mend <01:53.57> your <01:53.79> heart\n",
|
||||
"[01:55.24] <01:55.28> because <01:56.68> when <01:56.85> the <01:57.04> sun <01:57.35> shines <01:57.76> we <01:58.06> shine <01:58.43> together\n",
|
||||
"[01:59.24] <01:59.40> told <01:59.87> you <02:00.26> i'll <02:00.51> be <02:00.84> here <02:01.15> forever\n",
|
||||
"[02:02.15] <02:02.20> said <02:02.68> i'll <02:03.01> always <02:03.62> be <02:03.96> your <02:04.07> friend\n",
|
||||
"[02:04.64] <02:04.65> took <02:04.84> an <02:05.07> oath <02:05.29> gonna <02:05.56> stick <02:05.90> it <02:06.12> out <02:06.34> till <02:06.70> the <02:06.93> end\n",
|
||||
"[02:07.68] <02:07.70> now <02:08.06> that <02:08.19> it's <02:08.46> rainin' <02:09.12> more <02:09.42> than <02:09.69> ever\n",
|
||||
"[02:10.46] <02:10.47> know <02:10.84> that <02:11.19> we'll <02:11.52> still <02:11.87> have <02:12.25> each <02:12.46> other\n",
|
||||
"[02:13.21] <02:13.25> you <02:13.42> can <02:13.59> stand <02:13.97> under <02:14.30> my <02:14.84> umbrella\n",
|
||||
"[02:15.93] <02:15.97> you <02:16.14> can <02:16.28> stand <02:16.70> under <02:17.03> my <02:17.50> umbrella\n",
|
||||
"[02:18.74] <02:18.80> ella <02:19.47> ella <02:20.25> ae <02:20.92> ae <02:21.62> ae\n",
|
||||
"[02:22.12] <02:22.18> under <02:22.56> my <02:23.02> umbrella <02:24.30> ella <02:25.00> ella <02:25.75> ae <02:26.46> ae <02:27.15> ae\n",
|
||||
"[02:27.68] <02:27.74> under <02:28.08> my <02:28.52> umbrella <02:29.86> ella <02:30.47> ella <02:31.18> ae <02:31.90> ae <02:32.62> ae\n",
|
||||
"[02:33.14] <02:33.20> under <02:33.58> my <02:34.02> umbrella <02:35.34> ella <02:36.00> ella <02:36.78> ae <02:37.43> ae <02:38.15> ae\n",
|
||||
"[02:38.65] <02:38.78> ae <02:39.47> ae <02:39.86> ae\n",
|
||||
"[02:40.12] <02:40.15> you <02:40.42> can <02:40.70> run <02:41.12> into <02:41.71> my <02:42.25> arms\n",
|
||||
"[02:42.86] <02:42.96> it's <02:43.25> okay <02:43.78> don't <02:44.12> be <02:44.53> alarmed <02:45.31> come <02:45.78> into <02:46.70> me\n",
|
||||
"[02:47.14] <02:47.15> there's <02:47.50> no <02:47.75> distance <02:48.59> in <02:49.12> between <02:50.59> our <02:50.93> love\n",
|
||||
"[02:51.47] <02:51.50> so <02:51.81> go <02:52.08> on <02:52.24> and <02:52.50> let <02:52.84> the <02:53.20> rain <02:53.87> pour\n",
|
||||
"[02:54.64] <02:54.71> i'll <02:54.96> be <02:55.42> all <02:55.65> you <02:55.97> need <02:56.37> and <02:56.70> more\n",
|
||||
"[02:59.74] <03:01.43> because <03:02.80> when <03:02.96> the <03:03.15> sun <03:03.46> shines <03:03.87> we <03:04.18> shine <03:04.53> together\n",
|
||||
"[03:05.56] <03:05.59> told <03:05.97> you <03:06.37> i'll <03:06.64> be <03:07.03> here <03:07.28> forever\n",
|
||||
"[03:08.30] <03:08.34> said <03:08.78> i'll <03:09.18> always <03:09.75> be <03:10.21> your <03:10.34> friend <03:10.75> took <03:10.97> an <03:11.20> oath <03:11.43> gonna <03:11.78> stick <03:12.03> it <03:12.25> out <03:12.48> till <03:12.84> the <03:13.09> end\n",
|
||||
"[03:13.81] <03:13.86> now <03:14.18> that <03:14.37> it's <03:14.75> rainin' <03:15.24> more <03:15.52> than <03:15.80> ever\n",
|
||||
"[03:16.58] <03:16.62> know <03:16.96> that <03:17.31> we'll <03:17.62> still <03:18.00> have <03:18.36> each <03:18.58> other\n",
|
||||
"[03:19.34] <03:19.36> you <03:19.53> can <03:19.70> stand <03:20.09> under <03:20.42> my <03:20.84> umbrella\n",
|
||||
"[03:22.08] <03:22.09> you <03:22.25> can <03:22.42> stand <03:22.84> under <03:23.15> my <03:23.58> umbrella\n",
|
||||
"[03:24.84] <03:24.92> ella <03:25.59> ella <03:26.37> ae <03:27.06> ae <03:27.74> ae\n",
|
||||
"[03:28.25] <03:28.31> under <03:28.68> my <03:29.12> umbrella <03:30.42> ella <03:31.12> ella <03:31.86> ae <03:32.53> ae <03:33.24> ae\n",
|
||||
"[03:33.75] <03:33.84> under <03:34.18> my <03:34.64> umbrella <03:35.92> ella <03:36.62> ella <03:37.37> ae <03:38.02> ae <03:38.75> ae\n",
|
||||
"[03:39.25] <03:39.31> under <03:39.70> my <03:40.14> umbrella <03:41.46> ella <03:42.12> ella <03:42.59> ae <03:42.90> ae <03:43.56> ae <03:44.25> ae <03:44.90> ae <03:45.59> ae\n",
|
||||
"[03:45.81] <03:45.98> it's <03:46.24> rainin' <03:46.90> raining\n",
|
||||
"[03:47.86] <03:48.00> oh <03:48.28> baby <03:48.70> it's <03:48.96> rainin' <03:49.64> raining\n",
|
||||
"[03:50.65] <03:50.70> baby <03:51.40> come <03:51.78> here <03:52.50> to <03:52.75> me\n",
|
||||
"[03:53.25] <03:53.56> come <03:53.70> here <03:54.12> to <03:54.40> me\n",
|
||||
"[03:54.48] <03:54.64> it's <03:55.58> raining <03:56.09> raining\n",
|
||||
"[03:56.40] <03:56.46> oh <03:56.86> baby <03:56.96> it's <03:57.24> raining <03:57.65> raining\n",
|
||||
"[03:57.80] <03:57.84> you <03:57.92> can <03:58.36> always <03:58.93> come <03:59.34> here\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from lsync import LyricsSync\n",
|
||||
"\n",
|
||||
"lsync = LyricsSync()\n",
|
||||
"words, lrc = lsync.sync(audio_path, lyrics_path)\n",
|
||||
"print(lrc)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Using Fine-tuned model\n",
|
||||
"\n",
|
||||
"Please update your model's id and path in lsync/config.py"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"YOUR_MODEL_ID = \"en-finetuned-base\"\n",
|
||||
"YOUR_MODEL_BLANK_ID = 29"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"[00:00.00] <00:27.53> you <00:27.85> have <00:28.17> my <00:28.35> heart\n",
|
||||
"[00:29.17] <00:29.46> and <00:29.60> we'll <00:29.96> never <00:30.28> be <00:30.58> worlds <00:30.98> apart\n",
|
||||
"[00:31.87] <00:32.35> maybe <00:33.03> in <00:33.35> magazines\n",
|
||||
"[00:35.07] <00:35.11> but <00:35.46> you'll <00:35.74> still <00:36.11> be <00:36.42> my <00:36.56> star\n",
|
||||
"[00:37.38] <00:37.88> baby <00:38.50> 'cause <00:38.93> in <00:39.20> the <00:39.39> dark\n",
|
||||
"[00:40.14] <00:40.61> you <00:40.92> can't <00:41.28> see <00:41.57> shiny <00:42.11> cars\n",
|
||||
"[00:42.96] <00:43.20> and <00:43.35> that's <00:43.67> when <00:44.02> you <00:44.34> need <00:44.70> me <00:44.88> there\n",
|
||||
"[00:45.60] <00:46.10> with <00:46.39> you <00:46.80> i'll <00:47.11> always <00:47.78> share\n",
|
||||
"[00:48.56] <00:49.17> because <00:50.56> when <00:50.71> the <00:50.92> sun <00:51.21> shines <00:51.62> we <00:51.93> shine <00:52.30> together\n",
|
||||
"[00:53.02] <00:53.34> told <00:53.70> you <00:54.06> i'll <00:54.39> be <00:54.71> here <00:55.03> forever\n",
|
||||
"[00:55.80] <00:56.10> said <00:56.52> i'll <00:56.84> always <00:57.48> be <00:57.80> your <00:57.98> friend\n",
|
||||
"[00:58.50] <00:58.52> took <00:58.74> an <00:58.92> oath <00:59.16> gonna <00:59.43> stick <00:59.74> it <00:59.89> out <01:00.20> till <01:00.53> the <01:00.75> end\n",
|
||||
"[01:01.28] <01:01.60> now <01:01.92> that <01:02.10> it's <01:02.32> rainin' <01:02.96> more <01:03.30> than <01:03.52> ever\n",
|
||||
"[01:04.06] <01:04.37> know <01:04.70> that <01:05.04> we'll <01:05.35> still <01:05.71> have <01:06.07> each <01:06.28> other\n",
|
||||
"[01:06.82] <01:07.14> you <01:07.29> can <01:07.46> stand <01:07.84> under <01:08.15> my <01:08.56> umbrella\n",
|
||||
"[01:09.54] <01:09.87> you <01:10.01> can <01:10.20> stand <01:10.57> under <01:10.90> my <01:11.29> umbrella\n",
|
||||
"[01:12.42] <01:12.65> ella <01:13.35> ella <01:14.04> ae <01:14.70> ae <01:15.40> ae\n",
|
||||
"[01:15.82] <01:16.04> under <01:16.40> my <01:16.78> umbrella <01:18.15> ella <01:18.85> ella <01:19.46> ae <01:20.20> ae <01:20.92> ae\n",
|
||||
"[01:21.32] <01:21.57> under <01:21.92> my <01:22.32> umbrella <01:23.68> ella <01:24.34> ella <01:24.98> ae <01:25.70> ae <01:26.32> ae\n",
|
||||
"[01:26.79] <01:27.07> under <01:27.43> my <01:27.82> umbrella <01:28.28> ella <01:29.18> ella <01:29.90> ae <01:30.06> ae <01:30.56> ae <01:31.21> ae <01:32.57> ae <01:33.29> ae\n",
|
||||
"[01:33.54] <01:33.57> these <01:33.98> fancy <01:34.48> things\n",
|
||||
"[01:35.21] <01:35.71> will <01:36.04> never <01:36.35> come <01:36.76> in <01:37.04> between\n",
|
||||
"[01:37.92] <01:38.42> you're <01:38.76> part <01:39.00> of <01:39.15> my <01:39.50> entity\n",
|
||||
"[01:40.71] <01:41.18> here <01:41.51> for <01:42.00> infinity\n",
|
||||
"[01:43.37] <01:43.93> when <01:44.12> the <01:44.29> war <01:44.64> has <01:44.98> took <01:45.28> it's <01:45.51> part\n",
|
||||
"[01:46.40] <01:46.70> when <01:46.85> the <01:47.01> world <01:47.35> has <01:47.71> dealt <01:48.06> it's <01:48.28> cards\n",
|
||||
"[01:49.35] <01:49.84> if <01:50.14> the <01:50.43> hand <01:50.84> is <01:50.98> hard\n",
|
||||
"[01:51.84] <01:52.37> together <01:52.90> we'll <01:53.21> mend <01:53.56> your <01:53.79> heart\n",
|
||||
"[01:54.96] <01:55.28> because <01:56.68> when <01:56.84> the <01:57.04> sun <01:57.34> shines <01:57.74> we <01:58.04> shine <01:58.42> together\n",
|
||||
"[01:59.46] <01:59.48> told <01:59.84> you <02:00.15> i'll <02:00.50> be <02:00.82> here <02:01.14> forever\n",
|
||||
"[02:01.90] <02:02.21> said <02:02.62> i'll <02:02.93> always <02:03.57> be <02:03.92> your <02:04.07> friend\n",
|
||||
"[02:04.60] <02:04.64> took <02:04.84> an <02:05.01> oath <02:05.28> gonna <02:05.62> stick <02:05.85> it <02:06.04> out <02:06.34> till <02:06.68> the <02:06.88> end\n",
|
||||
"[02:07.43] <02:07.71> now <02:08.06> that <02:08.24> it's <02:08.43> rainin' <02:09.08> more <02:09.42> than <02:09.64> ever\n",
|
||||
"[02:10.18] <02:10.50> know <02:10.81> that <02:11.18> we'll <02:11.50> still <02:11.84> have <02:12.21> each <02:12.40> other\n",
|
||||
"[02:12.96] <02:13.25> you <02:13.42> can <02:13.58> stand <02:13.97> under <02:14.28> my <02:14.65> umbrella\n",
|
||||
"[02:15.68] <02:16.00> you <02:16.12> can <02:16.24> stand <02:16.68> under <02:17.02> my <02:17.40> umbrella\n",
|
||||
"[02:18.53] <02:18.78> ella <02:19.46> ella <02:20.14> ae <02:20.81> ae <02:21.43> ae\n",
|
||||
"[02:21.93] <02:22.15> under <02:22.53> my <02:22.92> umbrella <02:24.28> ella <02:24.97> ella <02:25.65> ae <02:26.34> ae <02:27.06> ae\n",
|
||||
"[02:27.43] <02:27.71> under <02:28.03> my <02:28.43> umbrella <02:29.84> ella <02:30.46> ella <02:31.15> ae <02:31.80> ae <02:32.52> ae\n",
|
||||
"[02:32.90] <02:33.18> under <02:33.53> my <02:33.92> umbrella <02:35.30> ella <02:35.96> ella <02:36.68> ae <02:37.34> ae <02:38.06> ae\n",
|
||||
"[02:38.62] <02:38.68> ae <02:39.42> ae <02:39.75> ae\n",
|
||||
"[02:40.09] <02:40.15> you <02:40.42> can <02:40.68> run <02:41.08> into <02:41.68> my <02:42.08> arms\n",
|
||||
"[02:42.84] <02:42.92> it's <02:43.21> okay <02:43.75> don't <02:44.08> be <02:44.43> alarmed <02:45.30> come <02:45.70> into <02:46.65> me\n",
|
||||
"[02:47.40] <02:47.42> there's <02:47.64> no <02:47.70> distance <02:48.50> in <02:49.08> between <02:50.56> our <02:50.90> love\n",
|
||||
"[02:51.43] <02:51.47> so <02:51.81> go <02:52.08> on <02:52.20> and <02:52.47> let <02:52.81> the <02:53.18> rain <02:53.81> pour\n",
|
||||
"[02:54.53] <02:54.64> i'll <02:54.96> be <02:55.31> all <02:55.64> you <02:55.93> need <02:56.34> and <02:56.64> more\n",
|
||||
"[03:01.37] <03:01.42> because <03:02.80> when <03:02.96> the <03:03.14> sun <03:03.46> shines <03:03.86> we <03:04.15> shine <03:04.52> together\n",
|
||||
"[03:05.56] <03:05.58> told <03:05.93> you <03:06.28> i'll <03:06.62> be <03:06.96> here <03:07.28> forever\n",
|
||||
"[03:08.03] <03:08.34> said <03:08.74> i'll <03:09.08> always <03:09.70> be <03:10.03> your <03:10.20> friend <03:10.75> took <03:10.96> an <03:11.15> oath <03:11.40> gonna <03:11.68> stick <03:11.98> it <03:12.18> out <03:12.46> till <03:12.80> the <03:13.02> end\n",
|
||||
"[03:13.53] <03:13.84> now <03:14.18> that <03:14.36> it's <03:14.53> rainin' <03:15.18> more <03:15.52> than <03:15.75> ever\n",
|
||||
"[03:16.30] <03:16.62> know <03:16.93> that <03:17.28> we'll <03:17.59> still <03:17.96> have <03:18.31> each <03:18.50> other\n",
|
||||
"[03:19.06] <03:19.37> you <03:19.52> can <03:19.70> stand <03:20.08> under <03:20.37> my <03:20.78> umbrella\n",
|
||||
"[03:21.78] <03:22.09> you <03:22.25> can <03:22.42> stand <03:22.80> under <03:23.14> my <03:23.52> umbrella\n",
|
||||
"[03:24.65] <03:24.90> ella <03:25.59> ella <03:26.20> ae <03:26.92> ae <03:27.62> ae\n",
|
||||
"[03:28.21] <03:28.28> under <03:28.65> my <03:29.02> umbrella <03:30.40> ella <03:31.08> ella <03:31.75> ae <03:32.42> ae <03:33.14> ae\n",
|
||||
"[03:33.53] <03:33.81> under <03:34.14> my <03:34.53> umbrella <03:35.90> ella <03:36.58> ella <03:37.28> ae <03:37.92> ae <03:38.64> ae\n",
|
||||
"[03:39.02] <03:39.30> under <03:39.65> my <03:40.06> umbrella <03:41.42> ella <03:42.08> ella <03:42.80> ae <03:43.46> ae <03:44.14> ae <03:44.84> ae <03:45.21> ae <03:45.53> ae\n",
|
||||
"[03:45.80] <03:45.90> it's <03:46.24> rainin' <03:46.92> raining\n",
|
||||
"[03:47.46] <03:47.93> oh <03:48.28> baby <03:48.65> it's <03:48.96> rainin' <03:49.64> raining\n",
|
||||
"[03:50.24] <03:50.70> baby <03:51.36> come <03:51.75> here <03:52.43> to <03:52.75> me\n",
|
||||
"[03:52.92] <03:53.31> come <03:53.93> here <03:54.12> to <03:54.40> me\n",
|
||||
"[03:54.48] <03:54.52> it's <03:54.93> raining <03:55.53> raining\n",
|
||||
"[03:56.52] <03:56.58> oh <03:56.90> baby <03:57.02> it's <03:57.21> raining <03:57.92> raining\n",
|
||||
"[03:58.53] <03:58.59> you <03:58.90> can <03:59.30> always <03:59.46> come <03:59.59> here\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from lsync import LyricsSync\n",
|
||||
"\n",
|
||||
"lsync = LyricsSync(lang=YOUR_MODEL_ID, blank_id=YOUR_MODEL_BLANK_ID)\n",
|
||||
"words, lrc = lsync.sync(audio_path, lyrics_path)\n",
|
||||
"print(lrc)"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "lsync",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.7.12"
|
||||
},
|
||||
"orig_nbformat": 4
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
Executable
+259
@@ -0,0 +1,259 @@
|
||||
name: lsync
|
||||
channels:
|
||||
- pytorch
|
||||
- nvidia
|
||||
- defaults
|
||||
- conda-forge
|
||||
dependencies:
|
||||
- _libgcc_mutex=0.1=conda_forge
|
||||
- _openmp_mutex=4.5=2_kmp_llvm
|
||||
- autopep8=1.6.0=pyhd3eb1b0_1
|
||||
- backcall=0.2.0=pyhd3eb1b0_0
|
||||
- blas=1.0=mkl
|
||||
- bzip2=1.0.8=h7f98852_4
|
||||
- ca-certificates=2022.12.7=ha878542_0
|
||||
- colorama=0.4.6=pyhd8ed1ab_0
|
||||
- cuda-cudart=11.7.99=0
|
||||
- cuda-cupti=11.7.101=0
|
||||
- cuda-libraries=11.7.1=0
|
||||
- cuda-nvrtc=11.7.99=0
|
||||
- cuda-nvtx=11.7.91=0
|
||||
- cuda-runtime=11.7.1=0
|
||||
- decorator=5.1.1=pyhd3eb1b0_0
|
||||
- entrypoints=0.4=py37h06a4308_0
|
||||
- ffmpeg=4.3.2=h37c90e5_3
|
||||
- freetype=2.12.1=hca18f0e_1
|
||||
- gmp=6.2.1=h58526e2_0
|
||||
- gnutls=3.6.13=h79a8f9a_0
|
||||
- icu=72.1=hcb278e6_0
|
||||
- ipython_genutils=0.2.0=pyhd3eb1b0_1
|
||||
- ipywidgets=8.0.6=pyhd8ed1ab_0
|
||||
- jupyter_client=5.3.4=py37_0
|
||||
- jupyter_core=4.5.0=py_0
|
||||
- jupyterlab_widgets=3.0.7=pyhd8ed1ab_0
|
||||
- lame=3.100=h166bdaf_1003
|
||||
- ld_impl_linux-64=2.40=h41732ed_0
|
||||
- libcublas=11.10.3.66=0
|
||||
- libcufft=10.7.2.124=h4fbf590_0
|
||||
- libcufile=1.6.0.25=0
|
||||
- libcurand=10.3.2.56=0
|
||||
- libcusolver=11.4.0.1=0
|
||||
- libcusparse=11.7.4.91=0
|
||||
- libffi=3.4.2=h7f98852_5
|
||||
- libgcc-ng=12.2.0=h65d4601_19
|
||||
- libhwloc=2.9.1=hd6dc26d_0
|
||||
- libiconv=1.17=h166bdaf_0
|
||||
- libnpp=11.7.4.75=0
|
||||
- libnsl=2.0.0=h7f98852_0
|
||||
- libnvjpeg=11.8.0.2=0
|
||||
- libopus=1.3.1=h7f98852_1
|
||||
- libpng=1.6.39=h753d276_0
|
||||
- libsodium=1.0.18=h7b6447c_0
|
||||
- libsqlite=3.40.0=h753d276_0
|
||||
- libstdcxx-ng=12.2.0=h46fd767_19
|
||||
- libvpx=1.7.0=h439df22_0
|
||||
- libxml2=2.10.4=hfdac1af_0
|
||||
- libzlib=1.2.13=h166bdaf_4
|
||||
- llvm-openmp=16.0.1=h417c0b6_0
|
||||
- matplotlib-inline=0.1.6=py37h06a4308_0
|
||||
- mkl=2021.4.0=h8d4b97c_729
|
||||
- mkl-service=2.4.0=py37h402132d_0
|
||||
- mkl_fft=1.3.1=py37h3e078e5_1
|
||||
- mkl_random=1.2.2=py37h219a48f_0
|
||||
- ncurses=6.3=h27087fc_1
|
||||
- nettle=3.4.1=h1bed415_1002
|
||||
- numpy=1.21.5=py37h6c91a56_3
|
||||
- numpy-base=1.21.5=py37ha15fc14_3
|
||||
- openh264=2.1.1=h780b84a_0
|
||||
- openssl=3.1.0=h0b41bf4_0
|
||||
- parso=0.8.3=pyhd3eb1b0_0
|
||||
- pexpect=4.8.0=pyhd3eb1b0_3
|
||||
- pickleshare=0.7.5=pyhd3eb1b0_1003
|
||||
- pip=23.0.1=pyhd8ed1ab_0
|
||||
- ptyprocess=0.7.0=pyhd3eb1b0_2
|
||||
- pycodestyle=2.10.0=py37h06a4308_0
|
||||
- python=3.7.12=hf930737_100_cpython
|
||||
- python-dateutil=2.8.2=pyhd3eb1b0_0
|
||||
- python_abi=3.7=3_cp37m
|
||||
- pytorch=1.13.1=py3.7_cuda11.7_cudnn8.5.0_0
|
||||
- pytorch-cuda=11.7=h778d358_3
|
||||
- pytorch-mutex=1.0=cuda
|
||||
- readline=8.2=h8228510_1
|
||||
- six=1.16.0=pyh6c4a22f_0
|
||||
- sqlite=3.40.0=h4ff8645_0
|
||||
- tbb=2021.9.0=hf52228f_0
|
||||
- tk=8.6.12=h27826a3_0
|
||||
- toml=0.10.2=pyhd3eb1b0_0
|
||||
- torchaudio=0.13.1=py37_cu117
|
||||
- tornado=6.2=py37h5eee18b_0
|
||||
- tqdm=4.65.0=pyhd8ed1ab_1
|
||||
- typing_extensions=4.5.0=pyha770c72_0
|
||||
- wheel=0.40.0=pyhd8ed1ab_0
|
||||
- widgetsnbextension=4.0.7=pyhd8ed1ab_0
|
||||
- x264=1!161.3030=h7f98852_1
|
||||
- xz=5.2.6=h166bdaf_0
|
||||
- zeromq=4.3.4=h2531618_0
|
||||
- zlib=1.2.13=h166bdaf_4
|
||||
- zstd=1.5.2=h3eb15da_6
|
||||
- pip:
|
||||
- accelerate==0.19.0
|
||||
- aiohttp==3.8.4
|
||||
- aiosignal==1.3.1
|
||||
- antlr4-python3-runtime==4.9.3
|
||||
- anyio==3.6.2
|
||||
- appdirs==1.4.4
|
||||
- argon2-cffi==21.3.0
|
||||
- argon2-cffi-bindings==21.2.0
|
||||
- async-timeout==4.0.2
|
||||
- asynctest==0.13.0
|
||||
- attrs==22.2.0
|
||||
- audioread==3.0.0
|
||||
- beautifulsoup4==4.12.2
|
||||
- bleach==6.0.0
|
||||
- certifi==2022.12.7
|
||||
- cffi==1.15.1
|
||||
- charset-normalizer==3.1.0
|
||||
- click==8.1.3
|
||||
- cloudpickle==2.2.1
|
||||
- colorlog==6.7.0
|
||||
- cycler==0.11.0
|
||||
- cython==0.29.34
|
||||
- dali-dataset==1.1
|
||||
- datasets==2.12.0
|
||||
- debugpy==1.6.7
|
||||
- defusedxml==0.7.1
|
||||
- demucs==4.0.0
|
||||
- diffq==0.2.3
|
||||
- dill==0.3.6
|
||||
- docker-pycreds==0.4.0
|
||||
- docopt==0.6.2
|
||||
- dora-search==0.1.11
|
||||
- einops==0.6.0
|
||||
- evaluate==0.4.0
|
||||
- exceptiongroup==1.1.1
|
||||
- fastjsonschema==2.16.3
|
||||
- ffmpeg-python==0.2.0
|
||||
- filelock==3.11.0
|
||||
- fonttools==4.38.0
|
||||
- frozenlist==1.3.3
|
||||
- fsspec==2023.1.0
|
||||
- future==0.18.3
|
||||
- gitdb==4.0.10
|
||||
- gitpython==3.1.31
|
||||
- huggingface-hub==0.14.1
|
||||
- hydra-colorlog==1.2.0
|
||||
- hydra-core==1.3.2
|
||||
- hyperpyyaml==1.2.0
|
||||
- hypothesis==6.71.0
|
||||
- idna==3.4
|
||||
- importlib-metadata==4.13.0
|
||||
- importlib-resources==5.12.0
|
||||
- inquirerpy==0.3.4
|
||||
- ipykernel==6.16.2
|
||||
- ipython==7.34.0
|
||||
- jedi==0.18.2
|
||||
- jieba==0.42.1
|
||||
- jinja2==3.1.2
|
||||
- jiwer==3.0.1
|
||||
- joblib==1.2.0
|
||||
- jsonschema==4.17.3
|
||||
- julius==0.2.7
|
||||
- jupyter==1.0.0
|
||||
- jupyter-client==7.4.9
|
||||
- jupyter-console==6.6.3
|
||||
- jupyter-core==4.12.0
|
||||
- jupyter-server==1.24.0
|
||||
- jupyterlab-pygments==0.2.2
|
||||
- kiwisolver==1.4.4
|
||||
- lameenc==1.4.2
|
||||
- lazy-loader==0.2
|
||||
- librosa==0.10.0.post2
|
||||
- llvmlite==0.39.1
|
||||
- markupsafe==2.1.2
|
||||
- matplotlib==3.5.3
|
||||
- mistune==2.0.5
|
||||
- msgpack==1.0.5
|
||||
- multidict==6.0.4
|
||||
- multiprocess==0.70.14
|
||||
- musdb==0.4.0
|
||||
- museval==0.4.0
|
||||
- nbclassic==0.5.5
|
||||
- nbclient==0.7.3
|
||||
- nbconvert==7.3.1
|
||||
- nbformat==5.8.0
|
||||
- nest-asyncio==1.5.6
|
||||
- notebook==6.5.4
|
||||
- notebook-shim==0.2.2
|
||||
- num2words==0.5.12
|
||||
- numba==0.56.4
|
||||
- omegaconf==2.3.0
|
||||
- openunmix==1.2.1
|
||||
- packaging==23.1
|
||||
- pandas==1.3.5
|
||||
- pandocfilters==1.5.0
|
||||
- pathtools==0.1.2
|
||||
- pfzy==0.3.4
|
||||
- pillow==9.5.0
|
||||
- pipreqs==0.4.13
|
||||
- pkgutil-resolve-name==1.3.10
|
||||
- pooch==1.6.0
|
||||
- prometheus-client==0.16.0
|
||||
- prompt-toolkit==3.0.38
|
||||
- protobuf==4.22.3
|
||||
- psutil==5.9.4
|
||||
- pyaml==21.10.1
|
||||
- pyarrow==11.0.0
|
||||
- pycparser==2.21
|
||||
- pyctcdecode==0.5.0
|
||||
- pydub==0.25.1
|
||||
- pygments==2.15.0
|
||||
- pygtrie==2.5.0
|
||||
- pyparsing==3.0.9
|
||||
- pyrsistent==0.19.3
|
||||
- pytz==2023.3
|
||||
- pyyaml==6.0
|
||||
- pyzmq==25.0.2
|
||||
- qtconsole==5.4.2
|
||||
- qtpy==2.3.1
|
||||
- rapidfuzz==2.13.7
|
||||
- regex==2022.10.31
|
||||
- requests==2.28.2
|
||||
- responses==0.18.0
|
||||
- retrying==1.3.4
|
||||
- ruamel-yaml==0.17.24
|
||||
- ruamel-yaml-clib==0.2.7
|
||||
- scikit-learn==1.0.2
|
||||
- scipy==1.7.3
|
||||
- seaborn==0.12.2
|
||||
- send2trash==1.8.0
|
||||
- sentencepiece==0.1.99
|
||||
- sentry-sdk==1.19.1
|
||||
- setproctitle==1.3.2
|
||||
- setuptools==67.6.1
|
||||
- simplejson==3.19.1
|
||||
- smmap==5.0.0
|
||||
- sniffio==1.3.0
|
||||
- sortedcontainers==2.4.0
|
||||
- soundfile==0.12.1
|
||||
- soupsieve==2.4
|
||||
- soxr==0.3.5
|
||||
- speechbrain==0.5.14
|
||||
- stempeg==0.2.3
|
||||
- submitit==1.4.5
|
||||
- terminado==0.17.1
|
||||
- threadpoolctl==3.1.0
|
||||
- tinycss2==1.2.1
|
||||
- tokenizers==0.13.3
|
||||
- traitlets==5.9.0
|
||||
- transformers==4.29.1
|
||||
- treetable==0.2.5
|
||||
- urllib3==1.26.15
|
||||
- wandb==0.14.2
|
||||
- wcwidth==0.2.6
|
||||
- webencodings==0.5.1
|
||||
- websocket-client==1.5.1
|
||||
- xxhash==3.2.0
|
||||
- yarg==0.1.9
|
||||
- yarl==1.8.2
|
||||
- youtube-dl==2021.12.17
|
||||
- zipp==3.15.0
|
||||
Executable
+12
@@ -0,0 +1,12 @@
|
||||
import DALI as dali_code
|
||||
import os
|
||||
|
||||
dali_data_path = os.path.abspath('dataset/DALI/v1')
|
||||
path_audio = os.path.abspath('dataset/DALI/audio')
|
||||
dali_data = dali_code.get_the_DALI_dataset(dali_data_path, skip=[], keep=[])
|
||||
dali_info = dali_code.get_info(dali_data_path + '/info/DALI_DATA_INFO.gz')
|
||||
|
||||
|
||||
# Get Audio
|
||||
downloaded_audio = [os.path.splitext(x)[0] for x in os.listdir(path_audio)]
|
||||
errors = dali_code.get_audio(dali_info, path_audio, skip=downloaded_audio, keep=[])
|
||||
Executable
+46
@@ -0,0 +1,46 @@
|
||||
from .alignment import Aligner
|
||||
from .lyrics_processor import LyricsProcessor
|
||||
from .util import read_text, save_lrc, save_words, save_audio
|
||||
from .config import TARGET_SR
|
||||
from .lrc_formatter import LrcFormatter
|
||||
import os
|
||||
|
||||
class LyricsSync():
|
||||
def __init__(self, lang="en-US", blank_id=0) -> None:
|
||||
# Deferred so that merely importing this module doesn't pull in
|
||||
# transformers/demucs at module load time (those fail loudly if the
|
||||
# installed transformers version is broken or incompatible).
|
||||
from .voice_extractor import VoiceExtractor
|
||||
from .phoneme_recognizer import PhonemeRecognizer
|
||||
self.ve = VoiceExtractor()
|
||||
self.phone_rec = PhonemeRecognizer(lang=lang)
|
||||
self.lp = LyricsProcessor(lang=lang)
|
||||
self.blank_id = blank_id
|
||||
|
||||
def sync(self, audio_fn: str, text_fn: str, save=True, timing_precision: str = 'word'):
|
||||
audio_name = os.path.splitext(os.path.basename(audio_fn))[0]
|
||||
vocals = self.ve.extract_voice(audio_fn)
|
||||
if save:
|
||||
save_audio(vocals, audio_name, TARGET_SR)
|
||||
emission, _, _, frame_duration = self.phone_rec.recognize(vocals)
|
||||
|
||||
# Preprocessing & tokenization
|
||||
tokens = self.phone_rec.tokenize(text_fn)
|
||||
path = Aligner.align(emission, tokens, blank_id=self.blank_id)
|
||||
processed_lyrics = self.lp.process(text_fn)
|
||||
|
||||
# Always compute word-level for the basic LRC output
|
||||
words = self.lp.get_words_from_path(processed_lyrics, path, frame_duration)
|
||||
original_lyrics = read_text(text_fn)
|
||||
lrc = LrcFormatter.words2lrc(words, original_lyrics)
|
||||
if save:
|
||||
save_words(words, audio_name)
|
||||
save_lrc(lrc, audio_name)
|
||||
|
||||
# Return syllable tokens for ELRC when requested
|
||||
if timing_precision == 'syllable':
|
||||
return_words = self.lp.get_syllables_from_path(processed_lyrics, path, frame_duration)
|
||||
else:
|
||||
return_words = words
|
||||
|
||||
return return_words, lrc
|
||||
Executable
+56
@@ -0,0 +1,56 @@
|
||||
import torch
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass
|
||||
class Point:
|
||||
token_index: int
|
||||
time_index: int
|
||||
prob: float
|
||||
|
||||
|
||||
class Aligner():
|
||||
@staticmethod
|
||||
def align(emission, tokens, blank_id=0):
|
||||
trellis = Aligner.get_trellis(emission, tokens, blank_id=blank_id)
|
||||
path = Aligner.backtrack(trellis, emission, tokens, blank_id=blank_id)
|
||||
return path
|
||||
|
||||
@staticmethod
|
||||
def get_trellis(emission, tokens, blank_id=0):
|
||||
num_frame = emission.size(0)
|
||||
num_tokens = len(tokens)
|
||||
trellis = torch.empty((num_frame + 1, num_tokens + 1))
|
||||
trellis[0, 0] = 0
|
||||
trellis[1:, 0] = torch.cumsum(emission[:, blank_id], 0)
|
||||
trellis[0, -num_tokens:] = -float("inf")
|
||||
trellis[-num_tokens:, 0] = float("inf")
|
||||
|
||||
for t in range(num_frame):
|
||||
trellis[t + 1, 1:] = torch.maximum(
|
||||
# Score for staying at the same token
|
||||
trellis[t, 1:] + emission[t, blank_id],
|
||||
# Score for changing to the next token
|
||||
trellis[t, :-1] + emission[t, tokens],
|
||||
)
|
||||
return trellis
|
||||
|
||||
@staticmethod
|
||||
def backtrack(trellis, emission, tokens, blank_id=0):
|
||||
j = trellis.size(1) - 1
|
||||
t_start = torch.argmax(trellis[:, j]).item()
|
||||
|
||||
path = []
|
||||
for t in range(t_start, 0, -1):
|
||||
stayed = trellis[t - 1, j] + emission[t - 1, blank_id]
|
||||
changed = trellis[t - 1, j - 1] + emission[t - 1, tokens[j - 1]]
|
||||
prob = emission[t - 1, tokens[j - 1]
|
||||
if changed > stayed else 0].exp().item()
|
||||
path.append(Point(j - 1, t - 1, prob))
|
||||
if changed > stayed:
|
||||
j -= 1
|
||||
if j == 0:
|
||||
break
|
||||
else:
|
||||
raise Exception("Failed")
|
||||
return path[::-1]
|
||||
Executable
+10
@@ -0,0 +1,10 @@
|
||||
ORIGINAL_SR = 44100
|
||||
TARGET_SR = 16000
|
||||
SEG_HOP_LENGTH = 1
|
||||
|
||||
MODELS = {
|
||||
'en-US': 'facebook/wav2vec2-large-960h-lv60-self',
|
||||
'en-US-base': 'facebook/wav2vec2-base',
|
||||
'en-finetuned-base': './model/checkpoint-320000',
|
||||
'zh-CN': 'jonatasgrosman/wav2vec2-large-xlsr-53-chinese-zh-cn',
|
||||
}
|
||||
Executable
+84
@@ -0,0 +1,84 @@
|
||||
import numpy as np
|
||||
from typing import List
|
||||
from lsync.lrc_formatter import Word
|
||||
import dataclasses
|
||||
from tqdm import tqdm
|
||||
import os
|
||||
from lsync import LyricsSync
|
||||
|
||||
|
||||
def get_eval_df(dataset='jamendo'):
|
||||
import pandas as pd
|
||||
df = pd.DataFrame(columns=['name', 'audio', 'lyrics', 'annotation'])
|
||||
if dataset == 'jamendo':
|
||||
base = "dataset/jamendolyrics"
|
||||
songs = os.listdir(f'{base}/mp3')
|
||||
for song in songs:
|
||||
name, _ = os.path.splitext(song)
|
||||
df = df.append({
|
||||
'name': name,
|
||||
'audio': f"{base}/mp3/{name}.mp3",
|
||||
'lyrics': f"{base}/lyrics/{name}.raw.txt",
|
||||
'annotation': f"{base}/annotations/words/{name}.csv",
|
||||
}, ignore_index=True)
|
||||
return df
|
||||
|
||||
|
||||
class Eval:
|
||||
@staticmethod
|
||||
def eval_all(eval_df, lsync: LyricsSync):
|
||||
import pandas as pd
|
||||
record = pd.DataFrame(
|
||||
columns=['name', 'average_abs_err', 'percentage_of_correct_segments'])
|
||||
for idx, row in tqdm(eval_df.iterrows()):
|
||||
try:
|
||||
words, lrc = lsync.sync(row['audio'], row['lyrics'])
|
||||
abs_err, perc_corr_segs = Eval.evaluate(
|
||||
words, row['annotation'])
|
||||
record = record.append({
|
||||
'name': row['name'],
|
||||
'average_abs_err': abs_err,
|
||||
'percentage_of_correct_segments': perc_corr_segs
|
||||
}, ignore_index=True)
|
||||
|
||||
except Exception as e:
|
||||
# print(traceback.format_exc())
|
||||
continue
|
||||
record.to_csv("eval.csv", index=False)
|
||||
return record
|
||||
|
||||
@staticmethod
|
||||
def evaluate(words: List[Word], gt_path: str):
|
||||
import pandas as pd
|
||||
pred_df = pd.DataFrame([dataclasses.asdict(w) for w in words])
|
||||
truth_df = Eval.load_ground_truth_jamendo(gt_path)
|
||||
abs_err = Eval.average_abs_err(
|
||||
truth_df['start'].values, pred_df['start'].values)
|
||||
perc_corr_segs = Eval.percentage_of_correct_segments(
|
||||
truth_df['start'].values, pred_df['start'].values)
|
||||
return abs_err, perc_corr_segs
|
||||
|
||||
@staticmethod
|
||||
def average_abs_err(ref, pred):
|
||||
deviations = np.abs(ref - pred)
|
||||
return np.mean(deviations)
|
||||
|
||||
@staticmethod
|
||||
def percentage_of_correct_segments(ref, pred, window=0.3):
|
||||
deviations = np.abs(ref - pred)
|
||||
return np.mean(deviations <= window)
|
||||
|
||||
@staticmethod
|
||||
def load_ground_truth_jamendo(gt_path: str):
|
||||
import pandas as pd
|
||||
df = pd.read_csv(gt_path, skiprows=1, names=["start", "line_end"])
|
||||
return df
|
||||
|
||||
@staticmethod
|
||||
def load_ground_truth_hensen(gt_path: str):
|
||||
import pandas as pd
|
||||
data = np.genfromtxt(gt_path, delimiter='\t', dtype=[
|
||||
('start', float), ('end', float), ('label', 'U10')])
|
||||
filtered = data[data['label'] != 'PAU']
|
||||
df = pd.DataFrame(filtered)
|
||||
return df
|
||||
Executable
+47
@@ -0,0 +1,47 @@
|
||||
from dataclasses import dataclass
|
||||
from typing import List
|
||||
|
||||
def seconds_to_lrc(seconds, is_word = True):
|
||||
minutes = int(seconds // 60)
|
||||
seconds = seconds % 60
|
||||
hundredths = int((seconds % 1) * 100)
|
||||
seconds = int(seconds)
|
||||
formated = f"{minutes:02d}:{seconds:02d}.{hundredths:02d}"
|
||||
return f"<{formated}>" if is_word else f"[{formated}]"
|
||||
|
||||
|
||||
@dataclass
|
||||
class Word:
|
||||
label: str
|
||||
start: float
|
||||
end: float
|
||||
is_word_end: bool = True # False for intra-word syllables (e.g. "Hel" in "Hel-lo")
|
||||
|
||||
def __repr__(self):
|
||||
return f"{seconds_to_lrc(self.start)} {self.label}"
|
||||
|
||||
class LrcFormatter():
|
||||
@staticmethod
|
||||
def words2lrc(words: List[Word], original_lyrics: str, lang="en-US"):
|
||||
lrc = ""
|
||||
counter = 0
|
||||
word_end = None
|
||||
for line in original_lyrics.splitlines():
|
||||
if line == '': continue
|
||||
if word_end:
|
||||
lrc += f"\n{seconds_to_lrc(word_end, False)}"
|
||||
else:
|
||||
lrc += "[00:00.00]"
|
||||
if lang == "en-US":
|
||||
splitted_words = line.split(' ')
|
||||
elif lang == "zh-CN":
|
||||
splitted_words = line
|
||||
|
||||
for original_word in splitted_words:
|
||||
if original_word == '': continue
|
||||
word = words[counter]
|
||||
word.label = original_word
|
||||
lrc += f" {word}"
|
||||
word_end = word.end
|
||||
counter+=1
|
||||
return lrc
|
||||
Executable
+154
@@ -0,0 +1,154 @@
|
||||
from dataclasses import dataclass
|
||||
from .util import read_text
|
||||
from .lrc_formatter import Word
|
||||
|
||||
SEPARATOR = '|'
|
||||
|
||||
|
||||
@dataclass
|
||||
class Segment:
|
||||
label: str
|
||||
start: int
|
||||
end: int
|
||||
|
||||
|
||||
class LyricsProcessor():
|
||||
def __init__(self, lang: str = "en-US") -> None:
|
||||
self.lang = lang
|
||||
|
||||
def process(self, text_path):
|
||||
text = read_text(text_path)
|
||||
if self.lang == "en-US":
|
||||
return self.__process_en(text)
|
||||
elif self.lang == 'zh-CN':
|
||||
return self.__process_cn(text)
|
||||
elif self.lang.startswith("en") and self.lang.endswith("-base"):
|
||||
return self.__process_en(text, is_upper=False)
|
||||
|
||||
def __process_en(self, text: str, is_upper=True):
|
||||
# preprocessing
|
||||
if is_upper:
|
||||
text = text.upper()
|
||||
else:
|
||||
text = text.lower()
|
||||
text = text.replace(' ', '|')
|
||||
text = text.replace('\n', '|')
|
||||
text = text.replace('_', '\'')
|
||||
text = text.replace('’', '\'')
|
||||
return text
|
||||
|
||||
def __process_cn(self, text):
|
||||
# preprocessing
|
||||
return text
|
||||
|
||||
def get_words_from_path(self, text, path, frame_duration):
|
||||
# Skip repeating char
|
||||
i1, i2 = 0, 0
|
||||
segments = []
|
||||
while i1 < len(path):
|
||||
while i2 < len(path) and path[i1].token_index == path[i2].token_index:
|
||||
i2 += 1
|
||||
segments.append(
|
||||
Segment(
|
||||
text[path[i1].token_index],
|
||||
path[i1].time_index,
|
||||
path[i2 - 1].time_index + 1
|
||||
)
|
||||
)
|
||||
i1 = i2
|
||||
if self.lang.startswith("en"):
|
||||
return self.__merge_en(segments, frame_duration)
|
||||
elif self.lang == 'zh-CN':
|
||||
return [Word(s.label, s.start * frame_duration, s.end * frame_duration) for s in segments]
|
||||
|
||||
def get_syllables_from_path(self, text, path, frame_duration):
|
||||
"""Like get_words_from_path but returns syllable-level Word objects.
|
||||
|
||||
Intra-word syllables have is_word_end=False; the final syllable of each
|
||||
word has is_word_end=True. Falls back to word-level if pyphen is not
|
||||
installed.
|
||||
"""
|
||||
# Deduplicate path entries (same logic as get_words_from_path)
|
||||
i1, i2 = 0, 0
|
||||
segments = []
|
||||
while i1 < len(path):
|
||||
while i2 < len(path) and path[i1].token_index == path[i2].token_index:
|
||||
i2 += 1
|
||||
segments.append(
|
||||
Segment(
|
||||
text[path[i1].token_index],
|
||||
path[i1].time_index,
|
||||
path[i2 - 1].time_index + 1
|
||||
)
|
||||
)
|
||||
i1 = i2
|
||||
if self.lang.startswith("en"):
|
||||
return self.__syllabify_en(segments, frame_duration)
|
||||
elif self.lang == 'zh-CN':
|
||||
return [Word(s.label, s.start * frame_duration, s.end * frame_duration) for s in segments]
|
||||
|
||||
def __syllabify_en(self, segments, frame_duration, separator=SEPARATOR):
|
||||
"""Split word character groups into syllable-level Word objects via pyphen."""
|
||||
try:
|
||||
import pyphen
|
||||
dic = pyphen.Pyphen(lang='en_US')
|
||||
except ImportError:
|
||||
# Graceful fallback: return word-level alignment
|
||||
return self.__merge_en(segments, frame_duration, separator)
|
||||
|
||||
syllables = []
|
||||
i1, i2 = 0, 0
|
||||
while i1 < len(segments):
|
||||
if i2 >= len(segments) or segments[i2].label == separator:
|
||||
if i1 != i2:
|
||||
word_segs = segments[i1:i2]
|
||||
word_text = "".join([s.label for s in word_segs]) # uppercase
|
||||
|
||||
# pyphen works on lowercase; character count is identical
|
||||
positions = dic.positions(word_text.lower())
|
||||
|
||||
if not positions:
|
||||
# Single-syllable or unrecognised word — emit as one token
|
||||
syllables.append(Word(
|
||||
word_text,
|
||||
word_segs[0].start * frame_duration,
|
||||
word_segs[-1].end * frame_duration,
|
||||
is_word_end=True,
|
||||
))
|
||||
else:
|
||||
boundaries = [0] + list(positions) + [len(word_text)]
|
||||
for j in range(len(boundaries) - 1):
|
||||
s_char = boundaries[j]
|
||||
e_char = boundaries[j + 1]
|
||||
syl_segs = word_segs[s_char:e_char]
|
||||
if not syl_segs:
|
||||
continue
|
||||
is_last = (j == len(boundaries) - 2)
|
||||
syllables.append(Word(
|
||||
word_text[s_char:e_char],
|
||||
syl_segs[0].start * frame_duration,
|
||||
syl_segs[-1].end * frame_duration,
|
||||
is_word_end=is_last,
|
||||
))
|
||||
i1 = i2 + 1
|
||||
i2 = i1
|
||||
else:
|
||||
i2 += 1
|
||||
return syllables
|
||||
|
||||
def __merge_en(self, segments, frame_duration, separator=SEPARATOR):
|
||||
# Merge chars to word
|
||||
words = []
|
||||
i1, i2 = 0, 0
|
||||
while i1 < len(segments):
|
||||
if i2 >= len(segments) or segments[i2].label == separator:
|
||||
if i1 != i2:
|
||||
segs = segments[i1:i2]
|
||||
word = "".join([seg.label for seg in segs])
|
||||
words.append(Word(
|
||||
word, segments[i1].start * frame_duration, segments[i2 - 1].end * frame_duration))
|
||||
i1 = i2 + 1
|
||||
i2 = i1
|
||||
else:
|
||||
i2 += 1
|
||||
return words
|
||||
Executable
+73
@@ -0,0 +1,73 @@
|
||||
# Import directly from the concrete submodule to bypass transformers' lazy-loader,
|
||||
# which raises a misleading "Are this object's requirements defined correctly?" error
|
||||
# when the backing module can't be found via the top-level package.
|
||||
from transformers.models.wav2vec2.modeling_wav2vec2 import Wav2Vec2ForCTC
|
||||
from transformers.models.wav2vec2.processing_wav2vec2 import Wav2Vec2Processor
|
||||
from transformers.models.wav2vec2.tokenization_wav2vec2 import Wav2Vec2CTCTokenizer
|
||||
from transformers.models.wav2vec2.feature_extraction_wav2vec2 import Wav2Vec2FeatureExtractor
|
||||
from transformers.utils import logging
|
||||
import torch
|
||||
from .config import TARGET_SR, MODELS
|
||||
from .lyrics_processor import LyricsProcessor
|
||||
from .util import get_audio_segments
|
||||
|
||||
logging.set_verbosity_warning()
|
||||
|
||||
|
||||
class PhonemeRecognizer():
|
||||
def __init__(self, lang="en-US", device: str = None) -> None:
|
||||
model_id = MODELS[lang]
|
||||
if lang.endswith('-base'):
|
||||
tokenizer = Wav2Vec2CTCTokenizer(
|
||||
"vocab.json", unk_token="[UNK]", pad_token="[PAD]", word_delimiter_token="|")
|
||||
feature_extractor = Wav2Vec2FeatureExtractor(
|
||||
feature_size=1, sampling_rate=16000, padding_value=0.0, do_normalize=True, return_attention_mask=False)
|
||||
self.processor = Wav2Vec2Processor(
|
||||
feature_extractor=feature_extractor, tokenizer=tokenizer)
|
||||
else:
|
||||
self.processor = Wav2Vec2Processor.from_pretrained(model_id)
|
||||
if device is not None:
|
||||
self.device = device
|
||||
else:
|
||||
self.device = "cuda" if torch.cuda.is_available() else "cpu"
|
||||
self.model = Wav2Vec2ForCTC.from_pretrained(model_id).to(self.device)
|
||||
self.lp = LyricsProcessor(lang=lang)
|
||||
self.lang = lang
|
||||
|
||||
def recognize(self, vocals):
|
||||
self.duration_sec = 0
|
||||
segs = get_audio_segments(vocals)
|
||||
logits = self.__recognize(segs[0])
|
||||
for seg in segs[1:]:
|
||||
logits_seg = self.__recognize(seg)
|
||||
logits = torch.cat((logits, logits_seg), dim=1)
|
||||
|
||||
# normalize the emission with log_softmax() to avoid numerical instability by computing prob in log-domain
|
||||
emission = torch.log_softmax(logits, dim=-1)[0].cpu().detach()
|
||||
pred = self.get_pred(logits)
|
||||
transcription = self.get_transcription(pred)
|
||||
|
||||
# get real time
|
||||
frame_duration = self.model.config.inputs_to_logits_ratio / TARGET_SR
|
||||
return emission, pred, transcription, frame_duration
|
||||
|
||||
def tokenize(self, text_path):
|
||||
text = self.lp.process(text_path)
|
||||
return self.processor.tokenizer(text).input_ids
|
||||
|
||||
def __recognize(self, vocals):
|
||||
vals = self.processor(vocals, return_tensors="pt",
|
||||
padding="longest", sampling_rate=16000).input_values.to(self.device)
|
||||
self.duration_sec += vals.shape[1] / TARGET_SR
|
||||
with torch.no_grad():
|
||||
logits = self.model(vals).logits
|
||||
return logits.cpu().detach()
|
||||
|
||||
def get_labels(self):
|
||||
return [k for k, v in sorted(self.processor.tokenizer.get_vocab().items(), key=lambda x: x[1])]
|
||||
|
||||
def get_pred(self, logits):
|
||||
return torch.argmax(logits, dim=-1)
|
||||
|
||||
def get_transcription(self, pred):
|
||||
return self.processor.batch_decode(pred)
|
||||
Executable
+55
@@ -0,0 +1,55 @@
|
||||
import librosa
|
||||
import soundfile as sf
|
||||
from .config import ORIGINAL_SR, TARGET_SR
|
||||
from lsync.lrc_formatter import Word
|
||||
from typing import List
|
||||
import dataclasses
|
||||
import numpy as np
|
||||
|
||||
window_size = int(TARGET_SR * 15)
|
||||
hop_length = window_size
|
||||
|
||||
|
||||
def get_audio_segments(audio):
|
||||
"""Split audio to segments"""
|
||||
return librosa.util.frame(audio, frame_length=window_size, hop_length=hop_length, axis=0)
|
||||
|
||||
|
||||
def get_audio_segments_by_onsets(audio):
|
||||
onset_times = librosa.onset.onset_detect(
|
||||
y=audio, sr=TARGET_SR, backtrack=True)
|
||||
onset_boundaries = np.concatenate([onset_times, [len(audio)]])
|
||||
segments = []
|
||||
start_onset = 0
|
||||
for onset in onset_boundaries:
|
||||
segments.append(audio[start_onset:onset])
|
||||
return segments
|
||||
|
||||
|
||||
def read_text(text_path):
|
||||
with open(text_path, 'r') as file:
|
||||
data = file.read()
|
||||
return data
|
||||
|
||||
|
||||
def save_audio(audio, name, sr=ORIGINAL_SR):
|
||||
sf.write(f'output/vocals/{name}.wav', audio, sr)
|
||||
|
||||
|
||||
def save_audio(audio, name, sr=ORIGINAL_SR, out_path="output/vocals"):
|
||||
sf.write(f'{out_path}/{name}.wav', audio, sr)
|
||||
|
||||
|
||||
def save_audio_file(audio, path, sr=ORIGINAL_SR):
|
||||
sf.write(f'{path}.wav', audio, sr)
|
||||
|
||||
|
||||
def save_lrc(lrc: str, name: str):
|
||||
with open(f'output/lrc/{name}.lrc', 'w+') as fp:
|
||||
fp.write(lrc)
|
||||
|
||||
|
||||
def save_words(words: List[Word], name: str):
|
||||
import pandas as pd
|
||||
df = pd.DataFrame([dataclasses.asdict(w) for w in words])
|
||||
df.to_csv(f"output/words/{name}.csv", index=False)
|
||||
Executable
+43
@@ -0,0 +1,43 @@
|
||||
import torch
|
||||
import numpy as np
|
||||
from demucs.pretrained import get_model
|
||||
from demucs.apply import apply_model
|
||||
from demucs.separate import load_track
|
||||
import librosa
|
||||
from .config import TARGET_SR, ORIGINAL_SR
|
||||
|
||||
|
||||
class VoiceExtractor():
|
||||
def __init__(self) -> None:
|
||||
self.device = "cuda" if torch.cuda.is_available() else "cpu"
|
||||
self.model = self.__load_model()
|
||||
self.vocals_source_idx = self.model.sources.index("vocals")
|
||||
self.sr = self.model.samplerate
|
||||
|
||||
def extract_voice(self, audio_fn, post_process=True):
|
||||
audio = self.__load_audio(audio_fn)
|
||||
vocals = self.__extract_voice(audio)
|
||||
return vocals if not post_process else self.post_process(vocals)
|
||||
|
||||
def post_process(self, audio, sr=ORIGINAL_SR, target_sr=TARGET_SR):
|
||||
audio = librosa.resample(audio, orig_sr=sr, target_sr=target_sr)
|
||||
return audio
|
||||
|
||||
def __extract_voice(self, audio: np.array, shifts: int = 1) -> np.array:
|
||||
"""Extract mono vocals from the audio"""
|
||||
ref = audio.mean(0)
|
||||
audio = (audio - ref.mean()) / ref.std()
|
||||
with torch.no_grad():
|
||||
sources = apply_model(
|
||||
self.model, audio[None], device=self.device, shifts=shifts, split=True, overlap=0.25, progress=False)
|
||||
vocals = sources[0][self.vocals_source_idx]
|
||||
return vocals.cpu().numpy()[0, ...]
|
||||
|
||||
def __load_model(self):
|
||||
model = get_model(name="htdemucs", repo=None)
|
||||
model.to(self.device)
|
||||
model.eval()
|
||||
return model
|
||||
|
||||
def __load_audio(self, audio_fn: str) -> np.array:
|
||||
return load_track(audio_fn, 2, self.sr)
|
||||
Executable
+843
@@ -0,0 +1,843 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Lyrics-audio Alignment"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"audio_name = \"Kinematic_-_Peyote\"\n",
|
||||
"lyrics = f\"dataset/jamendolyrics/lyrics/{audio_name}.raw.txt\"\n",
|
||||
"audio = f\"dataset/jamendolyrics/mp3/{audio_name}.mp3\"\n",
|
||||
"lang = \"en-US\"\n",
|
||||
"blank_id = 0"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"True"
|
||||
]
|
||||
},
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"torch.cuda.is_available()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"\n",
|
||||
" <audio controls=\"controls\" >\n",
|
||||
" <source src=\"data:audio/mpeg;base64,SUQzBAAAAAB7N1RJVDIAAAAIAAAAUGV5b3RlAFRQRTEAAAALAAAAS2luZW1hdGljAFRSQ0sAAAADAAAANQBUQUxCAAAABwAAAEtpdGVzAFREUkMAAAAGAAAAMjAxNwBUQ09OAAAABgAAAFJvY2sAVENPTQAAAAsAAABLaW5lbWF0aWMAV1BVQgAAABcAAGh0dHA6Ly93d3cuamFtZW5kby5jb20AVFBVQgAAABgAAABodHRwOi8vd3d3LmphbWVuZG8uY29tAFRYWFgAAAAiAAAAVGFnZ2luZyB0aW1lADIwMTgtMDYtMjFUMjA6MjE6NDgAVEVOQwAAACYAAABKYW1lbmRvOmh0dHA6Ly93d3cuamFtZW5kby5jb218IExBTUUAV09BUwAAACcAAGh0dHA6Ly93d3cuamFtZW5kby5jb20vZW4vYWxidW0vMTY3ODE0AENPTU0AAAAoAAAAZW5nAGh0dHA6Ly93d3cuamFtZW5kby5jb20gY2Nfc3RhbmRhcmQAV09BRgAAACgAAGh0dHA6Ly93d3cuamFtZW5kby5jb20vZW4vdHJhY2svMTQ0MjAzMABXT0FSAAAAKAAAaHR0cDovL3d3dy5qYW1lbmRvLmNvbS9lbi9hcnRpc3QvNDU3NTczAFdDT1AAAAAvAABodHRwOi8vY3JlYXRpdmVjb21tb25zLm9yZy9saWNlbnNlcy9ieS1uZC8zLjAvAFRDT1AAAAAwAAAAaHR0cDovL2NyZWF0aXZlY29tbW9ucy5vcmcvbGljZW5zZXMvYnktbmQvMy4wLwBBUElDAAB1fgAAAGltYWdlL2pwZWcAAwD/2P/gABBKRklGAAEBAQBgAGAAAP/bAEMACAYGBwYFCAcHBwkJCAoMFA0MCwsMGRITDxQdGh8eHRocHCAkLicgIiwjHBwoNyksMDE0NDQfJzk9ODI8LjM0Mv/bAEMBCQkJDAsMGA0NGDIhHCEyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMv/AABEIAZABkAMBIgACEQEDEQH/xAAcAAACAgMBAQAAAAAAAAAAAAABAgADBAUGBwj/xABIEAABAwIFAgQCBwQIBAQHAAABAAIRAyEEBRIxQVFhBiJxgRORBxQyobHB0RUjQmIXJUNSY3KS4TNTgoMWJPDxVHN0hJOisv/EABoBAAMBAQEBAAAAAAAAAAAAAAABAgMEBQb/xAAzEQACAgEDAwIEBAYDAQEAAAAAAQIRAwQhMRJBUQVhEyKRoRQVQoEyUnGxwdEjM/Biov/aAAwDAQACEQMRAD8A8yJDLJdW43lR5JMSgD2C7jwewwsPKLp/iyRaAgNkIlAuS0Xi0IGEKck/aHojBJtsmiO47eLoEeaYsiAQgTvf0QLuBwBbBF+qpezlWu2F/VVumN0i4sqhEkhWN0gXSnS50C6C7YnG6bVAtdCQNht1UkaUDJqJP6KTKEwUOUDocNcidVhEfmlk8JgHG5meEEjtb5Z+aIA4sowEtMIxFt0ENg/BAt5tdPYJLh1kAmAE9kTGlQDnZBw6EwgYwkC1yiJFkAJ3KMbJk2QiN0pIA2TgXQIkQgExAb7WRceQiGjdAn2CQwAmZ5UjVY7og9d0YgkzEoAUtiCjFgU5uEoFoTCxS0E2FkSNJBiyYBE+ZqQWKY0Seqnw4EjhQjyxdWOEQBwExWBrYueiLqWzjuUW+Z0X9FaBNNogWO6pIhyaKHUxGkm25umDR/CgQC8oxcQkO3QNJDtkHiYBAurIkzKrIbr3QCYCIQcLC3omcLmEhNkikwNBmE5MEBIHFplM2SZKAYxMEogjV2SmwKkdSgRW0DVcz7Ji3eEDY3TDZIpgi6CsMgdVUQSbIEhqcB2ytkA8KunaQd08C6YpckLgUknpZORxKUNSBUCeigjpsrdIaEpgGEwsVzGkExdVwACQrHA3KTSQOqRSZVCEWVpaCO6GmAeqRXUVIjdMGydlYGEOm3umNyFYQEdZLy6VZpA2CmjfoUEOSIAAZ45hEAQEWiLIgAEi1kybF5uFIsbhQdUwaCZKBCgTMD1UAncwoYaRGx6JtIInjqgYhEcJ2tQiUzZBN9+iYmwabygWp4UOyBWLBSObq5Tg8IcbIGKIFpCVWbJHCDdIaBBTRYcIgR6qE9v9kICAQ2N0SI/JRv3p3AFMTe5GtBAlQgTA3RYYHoo6OEE9wRperHODaZHVIHRwpMwCqukIAEBGwko8oOjZSFgAO/ukIm6scbQgBHCBplUuJCBadlaRCERukVZVG9rpg4DhQXJCZwGmOUDsrmUwEndLpINkzR5ikMUwSLInUBNkQJKaQAgLFg3B3QY3k2TH7U8oxKKFZA0CyzqOW1sRlmLx7DTFLDFgeCbnUYELEa2Aup8N0sLV8OZyMbWNLDfEoGo5okwCTA7nZcutzywYuuPmP3aTLwwWSdPw/wCxom5VX/ZDszqOZToa9FMPJ1VTzpHIHVJj8BVy7EMo1H0XudTbUBpP1CCNp6rbeLvrAzVrHFgwQpA4IU/sClxHfr3W1ZlGXnxfg8IcKwYapgQ97BtqNMku9ZuuRa9xhHLPhqUqXhVtfn/Js8CcnCPZpfU4yDaVCNXAXU5b+zM4disubllHD6cO+pQrsLjUBaJ8xJvKmXZO+nkOHx1DL6GNxOKc6PrDwGUmAxYEiSTPotZeoxhanGnaVOu9tb3XZ/QhaZy3i7RyhaJQHNl2zckwZ8R5N8TCUqbcY13x8K1+trHgHaDsbGJVGVtyjNMTi8tfljKNKlSfUp16bj8bybyTYyO1lD9Uh09Si2krfGytrzvx2K/Cu6b9v8nGOBJsEul5XWUzgc4yTMyzLKGErYKm2tSfRLpc2YLXSb+qOTYau/DYd1TIsvdhHO0vxGKqaHVb3LSXD7grlr4xjJyjTi6abXi/IRwttJPk5ejh61cltKlUeWguIY0mANz6LKwuAq4nB4vEtqUmtwzA9we+HOBMeUcrscppYfKfEGe5fTw9KoylQqua+pJdpgHQb7XvytPhWYTMMuz/ABjsHh6bqNCk6k2mDppnVBIk8rL8wcrcVUfkp7fqfiyvgJUm99/sc5JjZSeJXSYxuC8PYTB0jl2HxmOxFAV6tTEyWsDtmtaCPmudr1mV8Q+qyhToNe6RTpzpb2E3XZg1LzfNGPy9n5Mp4lDl7mxqZLiWeG6OeF9L6rVxLsM1snWHATJG0LIq+Gcbhsvy3F4qrhsO7MXhuHoVHxULCY+IRENb3JXb+E6GSVvAOUOz/Etp4NmdPIYWkiq8tAaD0beT2C5TxCzNKvj91DPQHYg4unTc0CGGmXANDR/c0my2Um3RcsSUU/NGrxmSYzC5risuY0YqthidZws1GwBJdI4A5WBt7r1jJXYLJ/HHjDLsNlWD+BQwmIfTkOkMDG/u9/snnlaHw9lNLPsFmniGlkeCqPoPp0MNlzanw8PrIlz3FzpIAvE3JQsgnp7ez8/Y4Ijj3Uiy9Az3IMN+wsHmmZZfhMoxbMczD4mlgqrSypRd/aBoc6HC4SeNcD+z6FRmD8PZczJ3vb9SzTDS9zm/zPDiJN5BHomp2TLC4pvwcDEJwo5tpCUb+i0MHuPwg7ZTUOVNQ9UEoBiEo36IzuYQBvsky0GPVI4EBMQevslkmyBogsmdNo5SbutCd02B4QgfIdgAd+iDS7so0TcqwAIJboMRBBULZTQIuEQJ9k6IsQiNio25UfIlM0QEx9iEWQA9EdwoNrpCEIkwpF7JuUvKBoDrlITZWTYlVgE3QUgQRdMATfhAklFriAEhskBu/soG2Qc4ufKstAQJ7CRA5SgyfwTmI3QtEpjQHNM7o3tAurGN1kBQiOEiersKNWxW0weZUsPkGZYFzXmrinUnMLQIGkyZWrdIhOGrPJijlXTLs0/o7X9hxm4O0bVuZ0a3h9+W4xlR1SideDqtAOgn7TTP8JWyZ4iwg8R4bMfhVvg0sKKLmwNWoMLbX2lcyR3UJ6Lmn6fhld97/wD1z9a+potTNV7V9uDZZHmNHK8wqYis17muoVKY0ATLhAT4XMcvxGU0cszVmIaMO5zqGIoAEtDt2kHcStXIHYpHhpBNtrK8mjxzk57p7brtV/7YoZ5RVdjdZdjcsyrP8Hi8N9aqYajd5qNaHkwRYAxFwq8nzOhl+Y4nEVWVHNqUatMBgEy4WWlY/Q2CDZQVQTMQOqUtJjkpKVu1T/pv/spZZpquzs22V4+lgsBmlCo15di8MKLC0CAdU37LYYjM8ox1PAV8WzFDEYOiyl8Cnp+G/SbGf4Z5submCDIuniyWTRY5z63ad9n7V/YUdROKrt/5nRDP8H/4qx2YupVjhcZTdSe0QHgOaBI44WHQzDL8Dgs4weGOIqUsXSYyi6o0AyDJ1AH8FpnBANUr0/EtldVFc/y7ov8AETe79/vyb6pmmU5rgcNSzZuKpYnDU/hNr4YNcKjBsHA8jqtDiRhxiKn1X4nwNXk+LGqO8WlAtEmyGmYWuHTRxN9DdeOy/oTLK5rc31TO8PU8B4bIxTq/WaWPfinPIGgtLYjeZ9lm1vFGCzLKsoOY065zfLKzGtxDACK2Ha4HS6TOocH9VyxENSkQFv0oFlZ29Hxhl7PH+bZxUoYh2W5lTq0HsaAKrWPaBMTEiNpWHgM5yHBU8zyOtTx2LyDGOZUZUIa3EUqjRZ4H2T0jouWbAR0xexS6EP48jcY1nhelUw4y+rmtZvxQa7q1KnT/AHfRkE+bubLcVc6yPK/DOZ5Vkz8zxJzLQH/XGsYyiGmZAaTLuJXGwoJnfdPp8k/EatpD7hAjlEIkiFZiLpBRIhSegTdgUAVmwlANlMSNlARB4SGuCOASECY1bp3zaOUGsA3QNbIgZB4UO/UommSJC6nJvA2NznLqeOGLoUG1AS1r2uLine truncated
|
||||
" Your browser does not support the audio element.\n",
|
||||
" </audio>\n",
|
||||
" "
|
||||
],
|
||||
"text/plain": [
|
||||
"<IPython.lib.display.Audio object>"
|
||||
]
|
||||
},
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from IPython.lib.display import Audio\n",
|
||||
"from lsync.config import ORIGINAL_SR\n",
|
||||
"\n",
|
||||
"def show_audio(audio, sr):\n",
|
||||
" return Audio(audio, rate=sr, autoplay=False)\n",
|
||||
"\n",
|
||||
"show_audio(audio, ORIGINAL_SR)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Vocals Extraction"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from lsync.voice_extractor import VoiceExtractor\n",
|
||||
"\n",
|
||||
"ve = VoiceExtractor()\n",
|
||||
"vocals = ve.extract_voice(audio, post_process=False)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Extracted Vocals"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"\n",
|
||||
" <audio controls=\"controls\" >\n",
|
||||
" <source src=\"data:audio/wav;base64,UklGRvzJ9gBXQVZFZm10IBAAAAABAAEARKwAAIhYAQACABAAZGF0YdjJ9gDm//b/9v/1//L/9v/z//v/7//u/+//8P/l/+//7v/0//L/9v/w//T/9P8AAPr/+f/s//L/+//z/9r/9P/v/+H/3f/s/9//4v/o/+j/5//h/9//4f/Y//L/7//v/+//+f/o/+v/9v/+//X/9P/l/9v/1P/v/+r/6v/w/+7/2//e/+X/9P/y//n/4//t/+r/9f/u/+b/2//p/+z/8v/y/+r/2P/t/97/7//z/+//4P/n/+z//P/2/+3/5v/t//L/9f/1//z/8//2//H/CwD9/w0AAAD1//P/+f8BAAUA+P/4/+n/9f/i/+z/9v/7/+3/6P/i//j/6v/5/+7/7//r/+r/4//u/+b/8//t//n/+v8BAPj////q/+r/8/8JABMACwABAAwA7//7/wkACQADAAoAAAD//+7/9f/t/+3/8v8BAPr//P/z/wAA/f/0/+7/+f/p//r/7f/0/+//AAD8/wEA+P/9/wQACAAHAAsAAgAUAAoAGQAWABQABwAIAA8AFgATAA8AEQASAAoAGAATABMADgAaABQAHQAVABAAEgATAAkAEAAMABcABgAKAAoAEQALABAAAAD//wYABgABAAwACgALABAAEwASABoAEwAeABcAGQAbABUABwAMAAgADAAKACEAHgAcABYAGAAMABMAEgAeABAADwAHAA0ACwAOAAoADgATABEACwAPAAAA9//x/wIA9P/u//H/7v/h/+H/4P/p/+j/8v/j/93/1f/V/9b/0f/N/8z/zf/Z/9P/1P/U/9P/0v/k/9//7P/d/9f/0v/c/9z/0v/N/8P/zP/R/9T/1f/S/9j/0v/R/9b/3P/H/8P/wf/A/8H/xP/C/7z/uf+//8D/yf/J/8T/vf/F/7v/vP+0/8D/sP+u/7X/tP+r/7H/q/+t/6b/rP+l/7H/rP+o/6P/sf+q/63/rP+8/7D/sP+u/7X/qf+p/6v/p/+g/6v/nv+b/47/l/+Q/6X/mv+T/4T/ov+b/6H/kP+E/3n/f/91/33/ZP9v/13/Uf9O/1j/V/9U/1z/Qf9I/yr/Df8Y/+P+8v6Q/qj+of6Z/mH/cgBQ/mf/if8YAEkAfQC4AK4A2QCgAKsAggCCAG4AcgB2AH8AfAB6AH8AjgCJAJIAkgCSAIIAcQBkAG8AagB2AGoAZgBgAHcAfwCAAH8AgACEAIcAhQCEAIEAfwBxAHgAegBxAHAAbQB3AHQAegB4AIIAcQB7AHQAdgBzAHkAdgB3AHkAewBwAIQAZgBfAF0AWgBKAD0APQA9ADAAOwA2AEEANAA2ACoAKgAwAD4AOgA9ADkAMAA4AEoANgA+ADoAQAA5AEEAQQBLADkAPAAtADMAMQAvABwAIgAfACMAIAAkABUAHgAWACAAHgAlAB4AJQAbACcAIgAdABMAEQAYAB4AFwAZABMACwAGAA4A+P8AAAEACwD2//T/8f/y/+X/9//n/+z/4v/y/+v/9f/l/+7/8P8AAPj//f/y//n/6f/6//r/BgD2/wAA9/8AAOz////z//r/7/8AAPL/AgD4/wsA+f8HAPz/BAD1/woA9//7/+3/+//s////5P/z/9z/6P/l/+z/2f/s/9z/6v/j//X/6f/1/+T//P/t//7/+f8CAPD//P/8/wIA8P/4/+j/+//t//n/7v/9/+z/9P/0/wMA8//2//H/AAD7/wYA9v8EAPn/AAD6/wAA9//5//D/+P/p/+//8////+T/8P/m/+r/6P/u//T/7P/v//z/3v/f/9P/yf/E/+L/4/8EAPL/+//t//3/9f8CAAMADAADAA8AAgAPAPr/CgD6/wcA+/8CAAAABADs//f/7P8CAPX//P/x//n/5v/s/97/6v/k/+7/1//d/9T/4f/g/+7/5P/r/+D/7v/j//T/7f/x/9z/6P/p//n/7f/w/9//9f/x//X/9v/5/+z/+v/2/wAA/P8BAPD/9//w//v/8f/z/+b/8//y////+v/4//L/+//x//3//P8FAPT/9//r/+7/7v8BAPH/+f/y//z/7v/8/wIAAQD5/wMA9//+//v/AwABAAQAAAAMAPr/CAAKAAQA/P8GAAcAAwD5/wEA/P8CAPr/AgD5/wEA/v8DAPv/AQD6/wMA+P8AAPj/BAD+/wYA+f8FAAUADQAAAAwABgASAAMADgAEABEABQARAAIADAABAAsAAAALAP3/BAD4//7/8P8FAP3/AgD3/wwA+v8FAP7/CAAAAAoA/v8FAP7/DQD9/wYA//8KAP//CgAFABIABgANAAQADAAAAAIA/f8EAP7/BgD0//3/+/8DAPn/AQD5/wAA+/8FAP7/BAD7/wYA/P8EAP7//v/2/wEA9P/8/+///P/4/wMA/P8BAPD/9P/z//3/8/8AAPj//v/5/wYA9v8EAAUACgD9/wgAAQAJAPz/BAD4/wcA+/8KAP//CAD5/wsADQAUAAsACwD//wsA//8HAPj/AwDz////9//8//D/AwD9/wYA+P8FAPn/AgD0/wMA9P8HAAAACwD5/wEA+/8FAP3/BQD5/wEA8f8CAPf/AAD1//7/9v8CAPb////3/wAA+P8CAPb/AQDv////9/8AAPD/+//z//z/9P/6//L//f/1/wAA9P8DAPr/BgD5/wUA//8BAPn/BgD//wcA/v8GAP3/BgD8/wUA/P8KAP//BgAAAAkAAAAJAP3/BQD5/wUA+f8DAPr/AAD4/wAA+f8FAPv/BgD7/wIA+f8FAPT/BAAAAAcA+/8GAP3/CwD//wkA/v8JAP7/CAD7/wgA+v8HAPz/BQD2/wQA+v8EAPf/AgD4/wcA9/8GAAAACgD6/w4A//8NAP3/DAD+/wkA+P8IAPX/BAD+/wsA/f8IAAAACgD6/woA/f8KAPv/BQD1/wQA9/8GAPr/CQD6/wYA+v8CAPv/BwD4/wIA9/8FAPX/AAD1/wEA+P8EAPr/BgDz/wUA/P8HAPz/BwD7/wMA+f8GAPz/BgD6/wYA+f8FAP3/DAD+/wkA9/8GAPr/AwD5/wMA9f8AAPn/BgD4/wMA8f8AAPP/AADy//v/8v/8//H//P/u//r/7f/4/+//+//s//n/6f/3/+7/9v/t//n/7//9//P//P/4/wIA9f8CAOb/9f8DAPv/7v/5/+z/9//u/wEA9f8AAPP/AADy//n/8//8//D//P/0//7/8v8AAPT////1/wMA9/8BAPb/AAD0/wIA9v/+//T//v/y/wAA9/8CAPb/AQDy/wEA9v8AAPb/AwD1/wEA+P8CAPf/AgD6/wUA+f8EAPf/BAD5/wYA+P8AAPr/AgD2/wEA+f8CAPj/AQD1/wEA9/8DAPX/AQD4/wIA+f8AAPb//f/w////8/////j/AwD1/wMA+f8AAPb/AADy/wAA8/8AAPX/AAD3/wMA+v8HAPz/BQD4/wIA+P8CAPT/AADx/wAA8/////X/AAD2/wEA+f8EAPX/AAD0/wEA9v8BAPT/AwD5/wYA+v8FAPj/BgD8/wYA9/8GAPb/AwD0/wEA8P/5/+r/+v/u/wAA8P////D/AADw//v/8P8AAO///v/y//7/8f8CAPT/BAD4/wcA8/8GAPf/AwDy/wMA9P////T/AADs//z/8f8AAPD/AAD0/wgA+P8IAAEABQD7/wkA/v8KAP7/EQAAAA0A/v8SAP7/CwD+/wsA+/8HAPj/CAD1/wgA+f8HAPr/DgD8/w0A//8PAPn/BQD1/wIA8f8JAOr/+v/v/wAA9f8DAPj/BwD4/xIA8f8bAPr/DwD2/wMA7f8FAPP/AAAAAA8ADAArABgARQAiACkACQAhAPf/0//T/6H/mf+f/5P/t/+u/8//zf/o/93/9//w//v/7//z/+j/+//s////7v/9//D//f/z/wIA/v8BAPf/DQD7/wQA/v8KAPr/CAD8/woA+P8LAPz/CQD//wwAAAAMAAAACwD9/woA/v8GAPj/CQD5/wQA9/8IAPn/BgD1/woA+/8CAPb/BgD0/wcA+/8LAPv/CwAAAA0A+v8JAPr/BQD3/wcA+P8IAPr/CgD2/wQA+P8FAPf/BwD4/wQA+P8HAPX/CAD4/wQA+P8HAPr/BgD6/wgA//8OAP//CQD5/woA+v8HAPn/CQD4/wYA+/8IAPr/BQD2/wQA+f8EAPj/CAD5/wYA/P8IAPr/BQD7/wcA+v8IAPv/CAD+/wkA/v8LAPv/BwD6/wYA+P8DAPX/BQD2/wYA+P8EAPn/BAD7/woA+v8HAPj/BAD5/wgA9v8HAPr/CAD9/w0A/f8LAPv/CAD6/woA+/8LAPr/CQD7/wsA/f8KAP3/CwAAAA4A/f8PAP7/CQD6/woA+/8KAPv/CAD5/wcA/P8JAPr/CQD6/wcA9/8IAPn/AwD7/wcA+v8GAPn/BgD5/woA+f8JAPv/BAD7/wUA+v8EAPb/AAD1/wIA9/8EAPf/CAD4/wMA+v8EAPn/BgD5/wIA9/8EAPj/AgD3/wYA+f8DAPb/AwDs//X/AAADAPb//v/v//z/8P/9//D/AgDz/wMA9/8BAPr/BAD4/wQA+P8FAPr/CQD5/wkA+P8IAPn/CgD9/wYA+v8HAPr/BwD2/wYA9v8HAPv/CgD6/wkA+v8IAPv/CAD3/wcA9v8IAPr/CQD6/wgA+/8IAPr/CAD6/wYA+P8FAPb/AwD7/wkA+f8IAPn/BwD6/wkA//8LAP3/DAD//wsA/P8LAP7/CAD5/wUA9/8EAPj/BgD5/wgA+v8IAPv/CAD8/wkA+f8JAPv/CAD7/wkA/v8JAP3/DAD7/wgA+v8GAPf/AwD4/wYA9v8GAPb/BwD4/wkA+/8JAP7/CgD+/wwA+/8LAP3/CQD6/wkA+v8HAPn/BwD6/wYA+P8IAPn/BwD3/wUA+/8HAPfLine truncated
|
||||
" Your browser does not support the audio element.\n",
|
||||
" </audio>\n",
|
||||
" "
|
||||
],
|
||||
"text/plain": [
|
||||
"<IPython.lib.display.Audio object>"
|
||||
]
|
||||
},
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from lsync.config import ORIGINAL_SR, TARGET_SR\n",
|
||||
"from lsync.util import save_audio\n",
|
||||
"\n",
|
||||
"save_audio(vocals, audio_name)\n",
|
||||
"show_audio(vocals, ORIGINAL_SR)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Vocals Post Processing "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"\n",
|
||||
" <audio controls=\"controls\" >\n",
|
||||
" <source src=\"data:audio/wav;base64,UklGRtCJWQBXQVZFZm10IBAAAAABAAEAgD4AAAB9AAACABAAZGF0YayJWQD0//P/9f/y/+v/7//y//X/+v/y/+3/5//h/+b/4//e/+//7//x//j/2f/s/+f/4v/z/+v/7f/h//L/4f/s/+j/7v/x/+v/+//z/wgA+f/8//3/6P/0/+z/7f/z/+f/6v/x/wAA7v/+/w4A+f8IAAMA8v/w//z/+v/1//D/8v/8//3/BwAJABYADwAPABMAEAAUABUAGAAOABAACwAOAAYAAgAKAA8AFwAZABkACAARAB8AEgAWABEACQAPABAAAwD0//X/4P/o/+b/1v/O/8//1P/T/+T/2f/X/8r/zv/V/9L/0P+8/8L/uf/D/8P/uv+3/63/sP+l/6v/pf+q/6z/tP+o/6n/nv+V/5T/lP+W/4n/cP9t/0z/U/9M/xr/8f57/jL/T//t/8cArQCVAGoAiwB+AKEAeQBzAGsAcgCDAIcAiQB9AHgAcwB9AHwAeQB5AH0AeQBlAEwAOQA7ADgALQA/ADcAQQA7AEMAQQA1ACoAIAAiABkAIQAjACIAFgAZABUAAgADAPr/7v/q/+v/7P/z//v/8f/8//3/9v/4//b/AQAAAAEA+f/1/+7/5P/l/+L/7f/t//f/+//7//X/8f/0//T/+f/3/wAA/P/+//T/7//z/+j/7P/x/+n/x//j//r/9P8FAAkABgAAAAEA+f/0//v/6//m/+L/1//p/+T/7v/n/+v/7//t//f/8/8AAPT/9f/t//j/9//2/wAA7v/1//b/9v8AAPz///8DAAQABQADAAAA/v//////AAD8////AAAFAAkACgAKAAsABgAGAP7/+v8AAAAAAwADAAMAAgAEAAkADAADAAEA/v/9//7//v8BAAAA///7//b//P/9//L/+//7/wAABAAFAAAAAAADAAQAEQAGAAIA/P/4//z/AAD9//v/AwD//wAA/v/7//v/+//8//3/+//5//j/9//3//n//v8AAAAAAAADAAAAAgADAAUAAgD///7//P8AAAAA/f8AAAEAAwAEAAIAAQAAAP///v/+/wAABAAFAAQAAAAAAAQAAwACAAIA/f8AAAAA//////7/+//9//////8DAP//AQAAAAEABAAAAP//+/////v/+f/3//b/9P/z//T/8P/z//T//P/5//b/+f/w//n/+v/3//b/9//5//r//f/7//z/+P/7//z/+v/8//z//f/+/////v/+//z//v/8//z//P/9//z/9v/7//3//f/6//r/+////wAA/v/7//n/+f/8//3/+//7//7/AAAAAAAA/v/6//L/9f/5//b/9//3//n//f////3//P/5//b/+P/8/wIAAgADAAcABwAGAAIAAAAAAAEABgAEAP3/+v/0//z/AAAFAAYA+v/7/wYAJwAsAA0Ayv+U/63/0f/v//T/8P/2//X//P8AAAEAAgACAAIAAwAGAAUAAwAAAAAA//8AAP7//v8CAAQAAwD//wAAAAD+////////////AAAAAAUAAwAAAAAAAAAAAP3/AAAAAAAAAAABAAIAAwABAP7//f//////AQAAAAAAAAADAAQAAQACAAEAAwADAAcABAACAAEAAAABAAAAAAAAAAAAAAACAAAAAAD9//v//////wAA///+//3/AAD5//n//v/0//n/+/////7///8BAAAAAQABAAAA/v8BAAEAAQAAAAAAAQAAAAEA/////wEAAAACAAUABAAEAAAA/v8AAAEAAgABAAEAAwAEAAIA///+//7/AAACAAQABAACAAAAAAAAAAAAAAAAAP///v8AAAAAAAD///3//f/+/wAA/f/+////AAAAAAAAAAACAAMABQAEAAMAAgABAAEAAQADAAIAAQAAAAAAAAD+//3/AAACAAQABQD//wMAAwAAAAIAAAAAAP7//f/9//7///////7//f/9//z/+//8//3//f///wAA///9//r/+//8//7//f/+//3//f/9//3//f/9//z/+//9//v/+//7//r/+//7//3//v///////f/9//z//f/8//3//v///////f/9//v//f/7//z//v//////AAABAP//AAAAAAAAAgAAAAIAAgABAAEAAwAEAAIAAgADAAUABQAEAAMAAAD+////+//7//j/9//8////AAAAAAAA/v/8//7//f/+////AAAAAP7//f/8//r/+v/7//v//P/9////AAD/////AAAAAP7/AAAAAP///v////3//f/+//3//f/+//////8AAP///v/9//3//P/8//v/+//7//v////+////AAD//////v/7//z//P/8/wAAAgADAAEA///+/////v///wAA/v8AAAAAAAD8//v//f8AAAgADAAOAAwADAAJAAUA+f/w/+L/1//m//3/BwAIAAIAAAAAAP//AAD+////AAACAAEAAgABAP//AAAAAAAAAQABAAAA/v/+//3//P/6//z//f/8//7//v/9//3/+//7//z/+//7//z//v/9//7//f/9//z//f///wAA///9//3//v8AAAAAAAD//wAAAAAAAAAA/////////////////f/+//3//v////////8AAAAAAAABAAAAAAD9////AQABAAIAAQAAAAMAAwAAAAAA+f/5//z/AAAAAAAAAAAAAAAAAAAAAP7//f/+///////+/wAAAAAAAAAA/v8AAAAAAAABAAAAAAAAAP7//P/9////AAAAAAAAAQAAAAAA/v/9//3///8AAAAAAAD///7//v/+//7//v/+//z//f/////////9//3//f/+//3//f/+//7///////////8AAAAAAQAAAAAAAAAAAAAAAQACAAEAAAD///7//f/7//3/AAABAAQABwAIAAcABQACAAEAAAD///7//v///wAAAQABAAAAAAAAAAAAAAAAAAEAAAABAAEAAQAAAP////8AAAAAAAAAAP///////wAA//////7//f/+//v//P/9//z//f/+//////8AAP///v/+//7//v/+//////8AAP7//f/8//v//P/7//3//v//////AAD///7////+/wAAAAAAAAAA///+////AAAAAAAAAAACAAQAAwACAAAA/f/7//r/+v/7////AAAAAAIAAAAAAP///P/8/////v8AAAAAAQAAAP7//f/7//r/+f/7//v//////wEAAAAAAAAAAgABAAAAAgAAAAAAAAAAAP//AAAAAAAAAAAAAAAAAAABAAAAAAAAAAAA//8AAP//AAD//wEAAgAAAAEAAQAAAAAAAAD+/wAAAAD//wIAAwAEAAEAAAABAAAA//////3//v/8//3/+v/5//7//v8FAAcABwAFAAQA+//9/+//9P/r/8v/3//0/wkADgALAAUABQAAAP/////9/wEAAgADAAMAAwABAAEAAAAAAAAAAgAAAP//////////+//9/////f/+//////////7//f///////f/+/wAAAAAAAAAAAAAAAAAAAQABAAIAAAAAAAAAAAAAAAEAAAAAAAAAAQABAAAAAAAAAAAA//////7//v///wAAAAABAAEAAAACAAEAAgABAAAAAAAAAAIAAwADAAMAAQABAAIAAAAAAPz/+f/8//7/AAAAAAAAAAD////////9//3//v/////////+/////v////7//P////7///8AAP/////+//z//P/+//7/AAD//wAAAAD///7//f/9//7/AAAAAAAA///+//7//v/+//7//v/9//z//v/9//7//f/8//3//f/+//3//v/+//7/////////AAAAAAEAAQAAAAAAAAABAAEAAwADAAEAAAAAAP///v/9/wAAAQADAAcABgAHAAgAAgABAAAAAAAAAAAAAAABAAMAAwACAAAAAQABAAAAAAACAAEAAgACAAIAAQAAAAAAAAAAAAAAAAAAAAAA//////////////3////+//3//v/9//3//f/+//////////3//v/+//7//v//////AAAAAP7//f/7//z//P/7//3//f/+//7////9//7//v/9/wAA//8AAP7//f/8//3/////////AAABAAIAAAAAAP3/+//6//v//P8AAAIAAQACAAEAAAAAAP7//P//////AAAAAAAAAAD///7//f/8//v//P/8//3//////wAA//8AAAAAAgAAAAIAAgAAAAAAAAAAAAAAAQAAAAEAAQABAAAAAwACAAEAAAAAAAAA/////wAAAAAAAAMAAQAAAAEAAAAAAAAA/f/+/////f///wEAAgACAAAAAAAAAAAAAAAAAP3//////wAA/P////7/AAAFAAUABgAFAAAAAgABAP7//P/S/8//4v/5/wQABAABAAAAAgD//wEA/f8AAAAAAwADAAUAAwACAAEAAQAAAAAAAQD//wAAAAAAAP3//P////////8AAAAA///+//3//f////3//f/+/wAA//8AAP//AAD+////AAAAAAEAAAAAAAAAAgABAAIAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAP7//////wAAAQABAAAAAQABAAEAAgABAAEA//8AAAMAAwADAAEA//8AAAAAAAD9//v//P///wAAAAAAAAAA///+/////v/9//7///8AAAAA/v////////8AAP3///8AAP//AAAAAP/////+//3////////////+/wAA///+//3//P/+////AAAAAAAA///+//3//v/+//7////+//7//////wAA/v/+//7////+//7////+////AAAAAAAAAAABAAIAAQAAAAAAAAABAAEAAgAAAAAAAAD//////f///wEAAQAEAAIAAQAGAAEAAAABAAAA//////////8BAAALine truncated
|
||||
" Your browser does not support the audio element.\n",
|
||||
" </audio>\n",
|
||||
" "
|
||||
],
|
||||
"text/plain": [
|
||||
"<IPython.lib.display.Audio object>"
|
||||
]
|
||||
},
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vocals = ve.post_process(vocals, sr=ORIGINAL_SR, target_sr=TARGET_SR)\n",
|
||||
"\n",
|
||||
"show_audio(vocals, TARGET_SR)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Phoneme Recognition"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Some weights of Wav2Vec2ForCTC were not initialized from the model checkpoint at facebook/wav2vec2-large-960h-lv60-self and are newly initialized: ['wav2vec2.masked_spec_embed']\n",
|
||||
"You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from lsync.phoneme_recognizer import PhonemeRecognizer\n",
|
||||
"\n",
|
||||
"phone_rec = PhonemeRecognizer(lang=lang)\n",
|
||||
"emission, pred, transcription, frame_duration = phone_rec.recognize(vocals)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"[\"BARR I'M ASLEEP AT THE WHEEL AND THERE'S A CAB IN THE HIWAY YET WIC SAT THROUGH THE NIFE LIKE A BEOT I MA TRAN HOW MUCH FETHER IS IT AGEIS SO EXQUISITE WE CAN'T STOP IT TIM NOW YOU DON'T WANT A CHANCE SO LOOK INRIGHT THOG YOUU GOS THAT YOU ER WIT DOAO NOW TE IS ON FIRE AND THE DESERT IS FREEZING WONDER HOW I'M LIVE WHEN I'M NOT EVEN BREATHING FATA IS IT ACHING SO EXQUISITE WE CAN'T STOP INTIDE NOW YOU DON'TANT TAN LOK A THE CLOTHES THAT YOU WEAR WITH DOWAT O STAR BUT THE PEOPLE BUT SAY WHAT THEY WAN TO THA'S TE TRUTH THAT IT IS WE DO RELYCA I\"]\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print(transcription)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Visualization of Phoneme"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"['<pad>', '<s>', '</s>', '<unk>', '|', 'E', 'T', 'A', 'O', 'N', 'I', 'H', 'S', 'R', 'D', 'L', 'U', 'M', 'W', 'C', 'F', 'G', 'Y', 'P', 'B', 'V', 'K', \"'\", 'X', 'J', 'Q', 'Z']\n",
|
||||
"torch.Size([8988, 32])\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAjIAAAD0CAYAAACB4szZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAABDoElEQVR4nO3de3wUVZ4F8NOPdCeQFwRJCBAIiIKigqAIgqIiDOooAwOKjInoqIOAPEQHXEFREVFnFZWHOg4MgyyKioCOsiygoosPUBkFiSg4oJIgYhIg5NHdtX+4NHTqhFy6A51OzvfzyedDKvW4dW9V5dI5/WuHZVkWRERERGKQM9oNEBEREQmXJjIiIiISszSRERERkZiliYyIiIjELE1kREREJGZpIiMiIiIxSxMZERERiVmayIiIiEjM0kRGREREYpYmMiJi891338HhcGD+/PnRbkqId955Bw6HA6+88kq0m3JSzJ8/Hw6HAxs2bKixffbu3RsdO3asdj12Ddx///1wOBwh67Vu3Ro33nhjjbVP5HhpIiP11uFfEuxr4sSJ0W6eSEzasmUL7r//fnz33XfRborUE+5oN0Ak2h544AFkZ2eHLDP5H2td1qpVKxw6dAhxcXHRbopEiek1kJeXB6fzyP+Jt2zZgqlTp6J3795o3br1CW6liCYyIujfvz+6du1qtG5paSk8Hk/Ig7sucjgciI+Pj3YzoubgwYNo2LBhje83EAigvLw8JvrW9Brwer0noTUiVavbT2ORCBzOYyxevBj33nsvmjdvjgYNGqC4uBj79u3DhAkTcNZZZyExMRHJycno378/Nm3aRPfx8ssvY+rUqWjevDmSkpLw+9//HkVFRSgrK8PYsWPRtGlTJCYmYvjw4SgrK7O1ZeHChejSpQsSEhLQuHFjXHfdddi1a1e15zB+/HikpaXh6A+5Hz16NBwOB5566qngsoKCAjgcDsyZMwcAz0fk5+dj+PDhaNGiBbxeL5o1a4ZrrrnG9ieEt956C7169ULDhg2RlJSEK6+8Eps3bzbpchQWFmLcuHFo3bo1vF4vWrRogZycHOzduzdkvUAggGnTpqFFixaIj4/HZZddhm+++SZknXXr1mHw4MHIysqC1+tFy5YtMW7cOBw6dChkvRtvvBGJiYn49ttvccUVVyApKQnDhg2rso2HcyJbt27FkCFDkJycjLS0NIwZMwalpaUh6zocDowaNQovvvgizjzzTHi9Xrz99tsAgM8++wz9+/dHcnIyEhMTcdlll+HDDz+kxywpKcFtt92GtLQ0JCcnIycnB7/88kvIOsuWLcOVV16JzMxMeL1etG3bFg8++CD8fj/d58aNG9GjRw8kJCQgOzsbc+fODfm5aU7q6IzM/PnzMXjwYADAJZdcEvxT7TvvvIPc3Fw0adIEFRUVtn307dsXp59++jGPI1IVvSIj9V5RUZHtF2WTJk2C/37wwQfh8XgwYcIElJWVwePxYMuWLXj99dcxePBgZGdno6CgAM8++ywuvvhibNmyBZmZmSH7mz59OhISEjBx4kR88803ePrppxEXFwen04lffvkF999/Pz788EPMnz8f2dnZmDJlSnDbadOmYfLkyRgyZAj++Mc/4qeffsLTTz+Niy66CJ999hlSU1OrPLdevXrhiSeewObNm4N/Llu3bh2cTifWrVuHO+64I7gMAC666KIq9zVo0CBs3rwZo0ePRuvWrbFnzx6sWrUKO3fuDP4J4R//+Adyc3PRr18/zJgxAyUlJZgzZw569uyJzz777Jh/ajhw4AB69eqFr776CjfddBPOPfdc7N27F8uXL8f3338fMiaPPPIInE4nJkyYgKKiIjz66KMYNmwYPvroo+A6S5YsQUlJCUaMGIG0tDR8/PHHePrpp/H9999jyZIlIcf2+Xzo168fevbsiccffxwNGjSosp2HDRkyBK1bt8b06dPx4Ycf4qmnnsIvv/yCBQsWhKy3Zs0avPzyyxg1ahSaNGmC1q1bY/PmzejVqxeSk5Nx9913Iy4uDs8++yx69+6Nd999F926dQvZx6hRo5Camor7778feXl5mDNnDv79738HJ8rAr5OIxMREjB8/HomJiVizZg2mTJmC4uJiPPbYYyH7++WXX3DFFVdgyJAhGDp0KF5++WWMGDECHo8HN910U7XnXpWLLroId9xxB5566incc8896NChAwCgQ4cOuOGGG7BgwQKsXLkSV111VXCb/Px8rFmzBvfdd1/Yx5V6zhKpp+bNm2cBoF+WZVlr1661AFht2rSxSkpKQrYtLS21/H5/yLIdO3ZYXq/XeuCBB4LLDu+jY8eOVnl5eXD50KFDLYfDYfXv3z9kH927d7datWoV/P67776zXC6XNW3atJD1vvjiC8vtdtuWV7Znzx4LgDV79mzLsiyrsLDQcjqd1uDBg6309PTgenfccYfVuHFjKxAIBM8FgDVv3jzLsizrl19+sQBYjz32WJXH2r9/v5WammrdcsstIcvz8/OtlJQU2/LKpkyZYgGwXnvtNdvPDrfrcH926NDBKisrC/585syZFgDriy++CC6rPGaWZVnTp0+3HA6H9e9//zu4LDc31wJgTZw48ZjtO+y+++6zAFhXX311yPLbb7/dAmBt2rQpuAyA5XQ6rc2bN4esO2DAAMvj8VjffvttcNmPP/5oJSUlWRdddFFw2eFrtEuXLiHXz6OPPmoBsJYtW3bM873tttusBg0aWKWlpcFlF198sQXA+stf/hJcVlZWZnXq1Mlq2rRp8DiVr4Gjz/1orVq1snJzc4PfL1myxAJgrV27NmQ9v99vtWjRwrr22mtDlv/nf/6n5XA4rO3bt9vaL2JCf1qSem/WrFlYtWpVyNfRcnNzkZCQELLM6/UGczJ+vx8///wzEhMTcfrpp+PTTz+1HSMnJyckNNmtWzdYlmX732+3bt2wa9cu+Hw+AMBrr72GQCCAIUOGYO/evcGvjIwMtGvXDmvXrj3muZ1yyilo37493nvvPQDABx98AJfLhbvuugsFBQXYtm0bgF9fkenZs6ftrbWHJSQkwOPx4J133rH9SeOwVatWobCwEEOHDg1pq8vlQrdu3apt66uvvopzzjkHv/vd72w/q9yu4cOHw+PxBL/v1asXAGD79u0hbT7s4MGD2Lt3L3r06AHLsvDZZ5/ZjjFixIhjtq+ykSNHhnw/evRoAMA///nPkOUXX3wxzjjjjOD3fr8f//3f/40BAwagTZs2weXNmjXD9ddfj/fffx/FxcUh+7j11ltDrp8RI0bA7XaHHOvo892/fz/27t2LXr16oaSkBFu3bg3Zn9vtxm233Rb83uPx4LbbbsOePXuwceNG4z44Hk6nE8OGDcPy5cuxf//+4PIXX3wRPXr0sAXuRUzpT0tS751//vnHDPuyB2wgEMDMmTMxe/Zs7NixIySHkJaWZls/Kysr5PuUlBQAQMuWLW3LA4EAioqKkJaWhm3btsGyLLRr14627fAvtwMHDuDAgQPB5S6XC6eccgqAX3/JH/6Ft27dOnTt2hVdu3ZF48aNsW7dOqSnp2PTpk24/vrrq+wDr9eLGTNm4M4770R6ejouuOACXHXVVcjJyUFGRgYABCdFl156Kd1HcnJylfsHgG+//RaDBg065jqHVe7PRo0aAUDIJGvnzp2YMmUKli9fbpt8FRUVhXzvdrvRokULo2MfVnlM2rZtC6fTacsMVb5+fvrpJ5SUlNBMSIcOHRAIBLBr1y6ceeaZVR4rMTERzZo1CznW5s2bce+992LNmjW2iVDl883MzLSFmU877TQAv2ZjLrjgAnLGkcvJycGMGTOwdOlS5OTkIC8vDxs3brTlc0SOhyYyItWo/GoMADz88MOYPHkybrrpJjz44INo3LgxnE4nxo4di0AgYFvf5XLRfVe13Pr/cG4gEIDD4cBbb71F101MTAQAPP7445g6dWpweatWrYK/5Hr27Innn38e27dvx7p169CrVy84HA707NkT69atQ2ZmJgKBQPBVjaqMHTsWv/3tb/H6669j5cqVmDx5MqZPn441a9agc+fOwfP+xz/+EZzcHM3trrnHTXX95vf7cfnll2Pfvn3485//jPbt26Nhw4b44YcfcOONN9rG6OhX2MJ1rFezTrTCwkJcfPHFSE5OxgMPPIC2bdsiPj4en376Kf785z/TazIazjjjDHTp0gULFy5ETk4OFi5cCI/HgyFDhkS7aRLDNJERCcMrr7yCSy65BC+88ELI8sLCwpBQaqTatm0Ly7KQnZ0d/B8zk5OTg549ewa/P/qX5+EJyqpVq/DJJ58Ei/1ddNFFmDNnTvB/5126dDFqz5133ok777wT27ZtQ6dOnfCXv/wFCxcuRNu2bQEATZs2RZ8+fcI61y+//PK4t2O++OILfP311/j73/+OnJyc4PLKfzaMxLZt20Jebfnmm28QCASqrZ1yyimnoEGDBsjLy7P9bOvWrXA6nbZX6rZt24ZLLrkk+P2BAwewe/duXHHFFQB+fXfczz//jNdeey0ksL1jxw7ahh9//NH2FvOvv/4aACKu/VLVhO6wnJwcjB8/Hrt378aiRYtw5ZVXBl9REwmHMjIiYXC5XCFvaQZ+fZfMDz/8UKPHGThwIFwuF6ZOnWo7nmVZ+PnnnwEAbdq0QZ8+fYJfF154YXC97OxsNG/eHE888QQqKiqCP+vVqxe+/fZbvPLKK7jggguO+YpJSUmJ7a3Fbdu2RVJSUvDt4v369UNycjIefvhh+hbbn3766ZjnOmjQIGzatAlLly61/azyuVfn8Cs2R29nWRZmzpx5XPs5llmzZoV8//TTTwP4tS5RdW3r27cvli1bFvKnoYKCAixatAg9e/a0/RnuueeeC+nTOXPmwOfzBY/Fzre8vByzZ8+mbfD5fHj22WdD1n322WdxyimnGE1oj+Xw5KiwsJD+fOjQoXA4HBgzZgy2b9+OP/zhDxEdT0SvyIiE4aqLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 640x480 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"labels = phone_rec.get_labels()\n",
|
||||
"\n",
|
||||
"print(labels)\n",
|
||||
"print(emission.shape)\n",
|
||||
"\n",
|
||||
"plt.imshow(emission[0:100].T)\n",
|
||||
"plt.title(\"Frame-wise char probability\")\n",
|
||||
"plt.xlabel(\"Time\")\n",
|
||||
"plt.ylabel(\"Labels\")\n",
|
||||
"# plt.yticks(range(len(labels)), labels)\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3. Alignment"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Generate trellis and find optimal path via backtracking"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"[10, 27, 17, 4, 7, 12, 15, 5, 5, 23, 4, 7, 6, 4, 6, 11, 5, 4, 18, 11, 5, 5, 15, 4, 7, 9, 14, 4, 6, 11, 5, 13, 5, 27, 12, 4, 7, 4, 19, 16, 13, 25, 5, 4, 10, 9, 4, 6, 11, 5, 4, 11, 10, 21, 11, 18, 7, 22, 4, 22, 5, 6, 4, 18, 5, 4, 6, 5, 7, 13, 4, 6, 11, 13, 8, 16, 21, 11, 4, 6, 11, 5, 4, 9, 10, 21, 11, 6, 4, 15, 10, 26, 5, 4, 7, 4, 23, 5, 22, 8, 6, 5, 4, 17, 10, 21, 13, 7, 10, 9, 5, 4, 11, 8, 18, 4, 17, 16, 19, 11, 4, 20, 16, 13, 6, 11, 5, 13, 4, 10, 12, 4, 10, 6, 4, 4, 7, 19, 11, 10, 9, 21, 4, 4, 12, 8, 4, 5, 28, 30, 16, 10, 12, 10, 6, 5, 4, 10, 4, 11, 8, 23, 5, 4, 18, 5, 4, 19, 7, 9, 4, 12, 6, 8, 23, 4, 10, 9, 4, 6, 10, 17, 5, 4, 4, 9, 8, 18, 4, 22, 8, 16, 4, 14, 8, 9, 27, 6, 4, 18, 7, 9, 6, 4, 6, 8, 4, 14, 7, 9, 19, 5, 4, 4, 27, 19, 7, 16, 12, 5, 4, 18, 5, 27, 13, 5, 4, 15, 8, 8, 26, 10, 9, 21, 4, 13, 10, 21, 11, 6, 4, 6, 11, 13, 8, 16, 21, 11, 4, 22, 8, 16, 4, 8, 11, 4, 8, 11, 4, 4, 7, 9, 14, 4, 6, 11, 5, 4, 19, 15, 8, 6, 11, 5, 12, 4, 6, 11, 7, 6, 4, 22, 8, 16, 4, 18, 5, 7, 13, 4, 4, 18, 5, 4, 14, 8, 9, 27, 6, 4, 17, 5, 7, 9, 4, 6, 8, 4, 12, 6, 7, 13, 5, 4, 4, 9, 8, 18, 4, 6, 11, 5, 4, 12, 26, 22, 4, 10, 12, 4, 8, 9, 4, 20, 10, 13, 5, 4, 7, 9, 14, 4, 6, 11, 5, 4, 14, 5, 12, 5, 13, 6, 4, 10, 12, 4, 20, 13, 5, 5, 31, 10, 9, 21, 4, 18, 8, 9, 14, 5, 13, 4, 11, 8, 18, 4, 10, 27, 17, 4, 7, 15, 10, 25, 5, 4, 4, 11, 5, 9, 4, 10, 27, 17, 4, 9, 8, 6, 4, 5, 25, 5, 9, 4, 24, 13, 5, 7, 6, 11, 10, 9, 21, 4, 11, 8, 18, 4, 17, 16, 19, 11, 4, 20, 16, 13, 6, 11, 5, 13, 4, 10, 12, 4, 10, 6, 4, 4, 7, 19, 11, 10, 9, 21, 4, 4, 12, 8, 4, 5, 28, 30, 16, 10, 12, 10, 6, 5, 4, 10, 4, 11, 8, 23, 5, 4, 18, 5, 4, 19, 7, 9, 4, 12, 6, 8, 23, 4, 10, 9, 4, 6, 10, 17, 5, 4, 4, 9, 8, 18, 4, 22, 8, 16, 4, 14, 8, 9, 27, 6, 4, 18, 7, 9, 6, 4, 6, 8, 4, 14, 7, 9, 19, 5, 4, 4, 27, 19, 7, 16, 12, 5, 4, 18, 5, 27, 13, 5, 4, 15, 8, 8, 26, 10, 9, 21, 4, 13, 10, 21, 11, 6, 4, 6, 11, 13, 8, 16, 21, 11, 4, 22, 8, 16, 4, 8, 11, 4, 8, 11, 4, 4, 7, 9, 14, 4, 6, 11, 5, 4, 19, 15, 8, 6, 11, 5, 12, 4, 6, 11, 7, 6, 4, 22, 8, 16, 4, 18, 5, 7, 13, 4, 4, 18, 5, 4, 14, 8, 9, 27, 6, 4, 17, 5, 7, 9, 4, 6, 8, 4, 12, 6, 7, 13, 5, 4, 8, 11, 4, 8, 11, 4, 4, 24, 16, 6, 4, 6, 11, 5, 4, 23, 5, 8, 23, 15, 5, 4, 22, 8, 16, 4, 26, 9, 8, 18, 4, 4, 6, 11, 5, 22, 4, 12, 7, 22, 4, 18, 11, 7, 6, 4, 6, 11, 5, 22, 4, 18, 7, 9, 6, 4, 6, 8, 4, 8, 11, 4, 8, 11, 4, 4, 27, 19, 7, 16, 12, 5, 4, 6, 11, 5, 4, 6, 13, 16, 6, 11, 4, 8, 20, 4, 10, 6, 4, 10, 12, 4, 4, 18, 5, 4, 14, 8, 9, 27, 6, 4, 13, 5, 7, 15, 15, 22, 4, 19, 7, 13, 5, 4]\n",
|
||||
"[(10, 'I'), (27, \"'\"), (17, 'M'), (4, '|'), (7, 'A'), (12, 'S'), (15, 'L'), (5, 'E'), (5, 'E'), (23, 'P'), (4, '|'), (7, 'A'), (6, 'T'), (4, '|'), (6, 'T'), (11, 'H'), (5, 'E'), (4, '|'), (18, 'W'), (11, 'H'), (5, 'E'), (5, 'E'), (15, 'L'), (4, '|'), (7, 'A'), (9, 'N'), (14, 'D'), (4, '|'), (6, 'T'), (11, 'H'), (5, 'E'), (13, 'R'), (5, 'E'), (27, \"'\"), (12, 'S'), (4, '|'), (7, 'A'), (4, '|'), (19, 'C'), (16, 'U'), (13, 'R'), (25, 'V'), (5, 'E'), (4, '|'), (10, 'I'), (9, 'N'), (4, '|'), (6, 'T'), (11, 'H'), (5, 'E'), (4, '|'), (11, 'H'), (10, 'I'), (21, 'G'), (11, 'H'), (18, 'W'), (7, 'A'), (22, 'Y'), (4, '|'), (22, 'Y'), (5, 'E'), (6, 'T'), (4, '|'), (18, 'W'), (5, 'E'), (4, '|'), (6, 'T'), (5, 'E'), (7, 'A'), (13, 'R'), (4, '|'), (6, 'T'), (11, 'H'), (13, 'R'), (8, 'O'), (16, 'U'), (21, 'G'), (11, 'H'), (4, '|'), (6, 'T'), (11, 'H'), (5, 'E'), (4, '|'), (9, 'N'), (10, 'I'), (21, 'G'), (11, 'H'), (6, 'T'), (4, '|'), (15, 'L'), (10, 'I'), (26, 'K'), (5, 'E'), (4, '|'), (7, 'A'), (4, '|'), (23, 'P'), (5, 'E'), (22, 'Y'), (8, 'O'), (6, 'T'), (5, 'E'), (4, '|'), (17, 'M'), (10, 'I'), (21, 'G'), (13, 'R'), (7, 'A'), (10, 'I'), (9, 'N'), (5, 'E'), (4, '|'), (11, 'H'), (8, 'O'), (18, 'W'), (4, '|'), (17, 'M'), (16, 'U'), (19, 'C'), (11, 'H'), (4, '|'), (20, 'F'), (16, 'U'), (13, 'R'), (6, 'T'), (11, 'H'), (5, 'E'), (13, 'R'), (4, '|'), (10, 'I'), (12, 'S'), (4, '|'), (10, 'I'), (6, 'T'), (4, '|'), (4, '|'), (7, 'A'), (19, 'C'), (11, 'H'), (10, 'I'), (9, 'N'), (21, 'G'), (4, '|'), (4, '|'), (12, 'S'), (8, 'O'), (4, '|'), (5, 'E'), (28, 'X'), (30, 'Q'), (16, 'U'), (10, 'I'), (12, 'S'), (10, 'I'), (6, 'T'), (5, 'E'), (4, '|'), (10, 'I'), (4, '|'), (11, 'H'), (8, 'O'), (23, 'P'), (5, 'E'), (4, '|'), (18, 'W'), (5, 'E'), (4, '|'), (19, 'C'), (7, 'A'), (9, 'N'), (4, '|'), (12, 'S'), (6, 'T'), (8, 'O'), (23, 'P'), (4, '|'), (10, 'I'), (9, 'N'), (4, '|'), (6, 'T'), (10, 'I'), (17, 'M'), (5, 'E'), (4, '|'), (4, '|'), (9, 'N'), (8, 'O'), (18, 'W'), (4, '|'), (22, 'Y'), (8, 'O'), (16, 'U'), (4, '|'), (14, 'D'), (8, 'O'), (9, 'N'), (27, \"'\"), (6, 'T'), (4, '|'), (18, 'W'), (7, 'A'), (9, 'N'), (6, 'T'), (4, '|'), (6, 'T'), (8, 'O'), (4, '|'), (14, 'D'), (7, 'A'), (9, 'N'), (19, 'C'), (5, 'E'), (4, '|'), (4, '|'), (27, \"'\"), (19, 'C'), (7, 'A'), (16, 'U'), (12, 'S'), (5, 'E'), (4, '|'), (18, 'W'), (5, 'E'), (27, \"'\"), (13, 'R'), (5, 'E'), (4, '|'), (15, 'L'), (8, 'O'), (8, 'O'), (26, 'K'), (10, 'I'), (9, 'N'), (21, 'G'), (4, '|'), (13, 'R'), (10, 'I'), (21, 'G'), (11, 'H'), (6, 'T'), (4, '|'), (6, 'T'), (11, 'H'), (13, 'R'), (8, 'O'), (16, 'U'), (21, 'G'), (11, 'H'), (4, '|'), (22, 'Y'), (8, 'O'), (16, 'U'), (4, '|'), (8, 'O'), (11, 'H'), (4, '|'), (8, 'O'), (11, 'H'), (4, '|'), (4, '|'), (7, 'A'), (9, 'N'), (14, 'D'), (4, '|'), (6, 'T'), (11, 'H'), (5, 'E'), (4, '|'), (19, 'C'), (15, 'L'), (8, 'O'), (6, 'T'), (11, 'H'), (5, 'E'), (12, 'S'), (4, '|'), (6, 'T'), (11, 'H'), (7, 'A'), (6, 'T'), (4, '|'), (22, 'Y'), (8, 'O'), (16, 'U'), (4, '|'), (18, 'W'), (5, 'E'), (7, 'A'), (13, 'R'), (4, '|'), (4, '|'), (18, 'W'), (5, 'E'), (4, '|'), (14, 'D'), (8, 'O'), (9, 'N'), (27, \"'\"), (6, 'T'), (4, '|'), (17, 'M'), (5, 'E'), (7, 'A'), (9, 'N'), (4, '|'), (6, 'T'), (8, 'O'), (4, '|'), (12, 'S'), (6, 'T'), (7, 'A'), (13, 'R'), (5, 'E'), (4, '|'), (4, '|'), (9, 'N'), (8, 'O'), (18, 'W'), (4, '|'), (6, 'T'), (11, 'H'), (5, 'E'), (4, '|'), (12, 'S'), (26, 'K'), (22, 'Y'), (4, '|'), (10, 'I'), (12, 'S'), (4, '|'), (8, 'O'), (9, 'N'), (4, '|'), (20, 'F'), (10, 'I'), (13, 'R'), (5, 'E'), (4, '|'), (7, 'A'), (9, 'N'), (14, 'D'), (4, '|'), (6, 'T'), (11, 'H'), (5, 'E'), (4, '|'), (14, 'D'), (5, 'E'), (12, 'S'), (5, 'E'), (13, 'R'), (6, 'T'), (4, '|'), (10, 'I'), (12, 'S'), (4, '|'), (20, 'F'), (13, 'R'), (5, 'E'), (5, 'E'), (31, 'Z'), (10, 'I'), (9, 'N'), (21, 'G'), (4, '|'), (18, 'W'), (8, 'O'), (9, 'N'), (14, 'D'), (5, 'E'), (13, 'R'), (4, '|'), (11, 'H'), (8, 'O'), (18, 'W'), (4, '|'), (10, 'I'), (27, \"'\"), (17, 'M'), (4, '|'), (7, 'A'), (15, 'L'), (10, 'I'), (25, 'V'), (5, 'E'), (4, '|'), (4, '|'), (11, 'H'), (5, 'E'), (9, 'N'), (4, '|'), (10, 'I'), (27, \"'\"), (17, 'M'), (4, '|'), (9, 'N'), (8, 'O'), (6, 'T'), (4, '|'), (5, 'E'), (25, 'V'), (5, 'E'), (9, 'N'), (4, '|'), (24, 'B'), (13, 'R'), (5, 'E'), (7, 'A'), (6, 'T'), (11, 'H'), (10, 'I'), (9, 'N'), (21, 'G'), (4, '|'), (11, 'H'), (8, 'O'), (18, 'W'), (4, '|'), (17, 'M'), (16, 'U'), (19, 'C'), (11, 'H'), (4, '|'), (20, 'F'), (16, 'U'), (13, 'R'), (6, 'T'), (11, 'H'), (5, 'E'), (13, 'R'), (4, '|'), (10, 'I'), (12, 'S'), (4, '|'), (10, 'I'), (6, 'T'), (4, '|'), (4, '|'), (7, 'A'), (19, 'C'), (11, 'H'), (10, 'I'), (9, 'N'), (21, 'G'), (4, '|'), (4, '|'), (12, 'S'), (8, 'O'), (4, '|'), (5, 'E'), (28, 'X'), (30, 'Q'), (16, 'U'), (10, 'I'), (12, 'S'), (10, 'I'), (6, 'T'), (5, 'E'), (4, '|'), (10, 'I'), (4, '|'), (11, 'H'), (8, 'O'), (23, 'P'), (5, 'E'), (4, '|'), (18, 'W'), (5, 'E'), (4, '|'), (19, 'C'), (7, 'A'), (9, 'N'), (4, '|'), (12, 'S'), (6, 'T'), (8, 'O'), (23, 'P'Line truncated
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from lsync.alignment import Aligner\n",
|
||||
"\n",
|
||||
"# Preprocessing & tokenization\n",
|
||||
"tokens = phone_rec.tokenize(lyrics)\n",
|
||||
"print(tokens)\n",
|
||||
"\n",
|
||||
"print([(c, labels[c]) for c in tokens])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Get trellis as the probability of a lyrics token occur at each time frame\n",
|
||||
"\n",
|
||||
"- Dimension: (num_frames + 1, num_tokens + 1)\n",
|
||||
"- Extra token for start-of-sentence\n",
|
||||
"- Extra frame for computation simplicity"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"torch.Size([8989, 731])\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAigAAACcCAYAAABVyVDPAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAABQSklEQVR4nO19e5RU1Z3ud07R3eKjGxHoBgXE6KAEfKFiazQzl740xswk0cxohhjiePWGgFExRjEPo6NidKLGd3Jnjc66K8qMMxpzNTrDoGg0HR+MRvFB1JBgIg1GQwMqLXTt+0f1ObUfv73PPlXnVV37W6tXV+3nb+86Vfs7v9fxGGMMDg4ODg4ODg4Fgp+3AA4ODg4ODg4OMhxBcXBwcHBwcCgcHEFxcHBwcHBwKBwcQXFwcHBwcHAoHBxBcXBwcHBwcCgcHEFxcHBwcHBwKBwcQXFwcHBwcHAoHEblLUAtKJfLePvtt7HXXnvB87y8xXFwcHBwcHCwAGMM27Ztw6RJk+D7Zh1JQxKUt99+G5MnT85bDAcHBwcHB4ca8NZbb2G//fYztmlIgrLXXnsBqCywvb09Z2kcasHnOs+uvAgY9EjVhBnuEELtn8+t3ZP2w/OqZb5X6RPUlfzKa3+4jKtj0ntyf/2U93x4fuZ5FWOy/B4AvKpsQkprn19HdTzGtYdfrWfhOjHcL7VViSD2UD+3J9ZL/+U1CGPqLiO+r27t0nutfBHfQbIfsRYm1wljDO+BP9xW2QPN+FnBk2QI3sufEaDKFqxd6sO3C8qVayCmjOFYGlmofRTm1IxRbcNI+bUy2Hz+w+/LO3Zgw1V/H57jJjQkQQl+2Nvb2x1BaUD07vVljGrdI3zvBYdss4EjHpX3nlLuyQTD96vEw/dCAsRKHLHhiYk8NveaRRxIdYMiJQKJgvC5Kwc03y54L5MS/ofSl9qnDWL/rA5x0GSE+Wo7oa08njwneeh4dFvDeBT0pEasZ75XKSLJCcI6nnCFh7lmjuw+T2I+/jqTiQbVV/oM+c+GBZe+BcGwgXFfCKJFEqOQTLHqmBHkiR8juLx4UiJfrrKc1Z+06EU3JEFxaGx4pdLwXb90QDcTCDJSKdYQEqEsICg+mKw94QkJRUR8qL8gaYHUnlTkFzQjXPuKvJKMvnigM9+r/ujxrz0vm8PM8pDXEQuKiFB35rZkQt4v6u48imDYgjpUhc/GSJYI+ahDUzdfmtCRx6g18f1NbT3xc697XSYSoPv8FRlpUhJJnDxJ2ymTLo8p7QX49o//cwTFITPM3+ecyouWUdoDuplAkhH+vUw4gv+eVyEmPHnhiEr4Axv+IFb7pq41CcDNXZGVIyWUZoTrp5pruMPXR5WUCIQGQh+5X+IwDK3ThAB6E4FW1S7MqRZqyYiJnNSxLaTmRz7cDOREPMgM7YK58vhZ0K1DQ6RMpIQfy0jWapBRV6bVkgCAxwxaFHp8hU6QnyVT59IgzpodQXHIBPMnfGXYZ8IXTRdAdnf0RQJFRsI6kYzw9YzTnmhJibS3ghZDnitFKNoSwoek0o4nIIYDnCMm6l2f+TDT+nDUAuWAIiY2yWI6qHRaIM14UQSlZpJmOZ9CToI63QFMEhTiM6/3AK8V8jogvlYOctCfJ0lMqLHrIotSgfazYcpnQ5l/wnE14ynzUqTEtJ6gLsZ30REUh9Qxf+JioFQSNQb83X+AZiMqBv+QUPMg1wVaFaNvh0RWONNKlhCIie+JhMQj2gGinGVGkxKP6OdD++Nocj6tFVH2f+t+hBYojsZDJQzy3hrkNIHoZ9QUDP+PWgf5WejGo8ZIE/JBLclIkRO5P6VNiiQH9UIgT1V9h9Z0w5cR41AEh5qTIqQU5M/Q3sDjCIpDyjhp33Ph+b6oMeBNFwGajZwAIqmgTDn8fxtS4ovjChoMQP8jkuLeB8TEGG3ja7QGo3ytpkS056vrS/wQ4GAmJ+bJtNoTwyFNj8M1iOhrfchHzim108kfQUQUmQyER2mbNigZbeWR6kPSLJObWj8fHXhfkmB8ak95QmLcf6a0IUkztR8UmdHIawNHUBzSxSjuEuMdOoHmJCXDYAYzDoDqXg2X01oRT2PSARfVU60Xf0gy2Hsi4iaUgfIjCWWr/pcP4mp4qse1AbQ/+tT4SSLmuGYTiUo6yL4aGUzkJpbGx9A2Un7twWWQR0N4Mjfv8LJI703aE0VDQBETeT2ypqNeeaNIXvDaRCpMa6U+vyhnWF25M/E45I2TDriw8qLEmXJkPwoezUhWZO2IVKcnIB5Qqvbhf2xYyQ9JiXxICw6oaSP8seOIicm5legDgAjb5MYTflCrAxnNDEmhhnFVk0GVZFH1xv6SLMa+NZAdHUhCJZRZyCOVURoYG1nSgNFnxgSKjAivCS1HrZD3ji/nCYlcxpMKknCI45Nr1vTXacqq43OEzEXxOOQNNqokFphMF80ImbQJ2pPgvy/WGXxK2LDZLDyshTwTwaEuHSgpg9KWkH4knEy8L4n2AA9/9D3lB19Qr6eFGvePXI/psIGmjpLHchyrsTTQEhL+NVVnmEfrn5Hjz4K1pkqu8yAmoOOdU7k2yhi1rpX/zOUyYS8NETam9jrZDOSE2RAYp0FxyAvzD76k8oLKEKvzs0CGd/ZFghwGzL2mfEoq5VD9SjyRmIjmD0/8UcyQEFKkhCQkOlkBoW2lvYaUePz/FNdYMzkR+1MHsfFw1sxrHIfoZ9ybmISCMidYm2c0xCYXs44MHTnUfD6UtsSokZAJZRJyagiJIB/RT+hDkQrD58EoIqMld1xbp0FxyA28SQeAbLrQ5uhoRlBRN1y5rCmhMq3KTqhyO8q0khUUYuLpNSTaAxyAbBLSHZJM52ybBiznociC1V2sYT7tHXxY76nyRRGciDmF9gaSYoqm0o1TBK0JEG0mI+WUiIGqISTa82PUuWbKbKP1Q5EJjE6OsL2BSJjkl7UonlTnCIpDHug99NtAadi0Q6nxTGaNAGmq5osIZU+Gy+Vsq4qPxTAxkZ5ZYzSrZAmOhLASFO0JZa4JYPQD8OTDrUp8Mr37jphLe9h5mnZR5AUG7UccEkTJZ2gn19lqUExQfXGi+2QKw0EvH/KUVkQxW8HQJqyrQY+imS+AQF6kdahjaMw6Np+LjpBQ5h4PzsTjkD16j/gOMEq88owExDnJCmSi8r56gCsJuyQtC+MIjJaUhGMR2pcMIETccAdRVTaVPEU6KerG8wyHbtKIOMQDRGs6qDLN52SY06x9MsugjGGYy3hnbrv/Uh/tgZ0ndPtJaEjEeqmfVEZpO5QxEpBR1yYRMsLPYSIkHtEelZ8xz2lQHLLEvKO/WyUnRht3cBBzZU3oJKsNJ/UIU4wceTN8QAcp3yvjcX2DMfmDnOeNWRIUkkh4CrFQ+3FvPGr9ELUmET+yqR1+EeNGaSlMppgoDYgyj2YPrElS1DyaA5nqH6md8YhD3YRMr9lgTjrhGWnC4vcmeC8QMUZ/riZiaYO42pGoPdckZKPbcn3kOT0mHgPS/J5f1gyqwhEUh5oxb84VlRe+H/3l0vlRBGhmDYqsHZGdXAMEP+xRmhIPxmgYYe6UwXxOTpmsyCDWC1DmIPEOnNZEJLoMRU4bGFPgGw6sKFJBlcceR6qzJQyRhIQgnaZDOXr8ml1Ia0JdWiLlemQqEaeu0RgaBWU+6bW142rcSB25n/C7w2lIfNXZ1uNIjOcB5RgmntgW/z/84Q/44he/iH322QejR4/GrFmz8Nxzz4X1jDF85zvfwcSJEzF69Gj09PTg9ddfF8Z47733sGDBArS3t2PMmDE466yzsH379riiOOSM8ig//GMlH2yU4c/nHhpX8sL3rDTc1/ea7g+lyh8reWCjpPpSkOp+WFMwykN5lLR3Pirf4GFfFOZ7YKNAtgv+yqXKH1+W2l9JnT/4C3/MpfVW+5rG5f4CM9KwNqWyxpw/2+HPNPhsws+IWFMoe6n6X2hf0uwFVy58niVxbHJ+ro1yPXiGP15ebhxFXklGcv0ewHxm/iuxbK5TTt7q98luT4X99StrwvBf2Ee5Rlm1XYlpx4v6o/ZMkC8k8az6F8xLtfP5eia+V/pV3ns+C802Xil4P/zfZ/BLXHlQl1Ym2T/96U84/vjj8Rd/8Rd4+OGHMX78eLz++uvYe++9wzbXXnstbrrpJvzzP/8zpk2bhm9/+9vo7e3FK6+8gt122w0AsGDBAmzcuBErV67Ezp07ceaZZ+Kcc87B3XffHUcchxzRc8JVoonBBjpNQTOC126Qd1Xc3sqmMY22RPFJAarvZZ+WtCHfefqGtRKaEh6KtoTQmshZTIXxsoSNZsCk7YiSP6Is0ofHVkYKnIw6rQjpC2EzfhE+O0oOroz+PERNBOlbQ2odUNF21LpO415qnFMpmcl6zXjBW/4zDjUjldd8XfhS7h/DxOMxxqzpzCWXXIKnnnoKP//5z8l6xhgmTZqECy+8EF//+tcBAAMDA+js7MRdd92F008/Ha+++ipmzJiBZ599FkcddRQA4JFHHsGnPvUp/P73v8ekSZOUcQcHBzE4OBi+37p1KyZPnoyBgQG0t7dbL9YhGcz95NWVFz5db5NrQWnThFyFJCOASEg0B7FASsDd3fFjy6YVCinue3VuiTjpXlPyyG1lchvsC7X+LGBLIjhoyYgFiRD6a9oazTCW14HxeiFJZo2ZUnWy5fB7YJvsLGxPfQ4hSYkgJUmsL8rZ1SYVvYG4eBHjCKSEN/OEdUwcZxjlD3bgN1++2ur8jqVB+elPf4re3l789V//NR5//HHsu++++OpXv4qzzz4bALB+/Xr09/ejp6cn7NPR0YE5c+agr68Pp59+Ovr6+jBmzJiQnABAT08PfN/H008/jc997nPKvMuXL8fll18eR1SHlPA/5i4HRlWvuFhaEOpurgmdZAGRjOjvRj2l3BgZY5kvJCvoonf45GyyTFpfCn4/YoyXJmL5isj9dOs0jGFFuqQ9I8cLine truncated
|
||||
"text/plain": [
|
||||
"<Figure size 640x480 with 1 Axes>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"trellis = Aligner.get_trellis(emission, tokens, blank_id=blank_id)\n",
|
||||
"print(trellis.shape)\n",
|
||||
"\n",
|
||||
"plt.imshow(trellis[1:, 1:][0:3200].T, origin=\"lower\")\n",
|
||||
"# plt.yticks(range(len(labels)), labels)\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Backtrack to find the most likely alignment path"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"[Point(token_index=0, time_index=1139, prob=0.9958117008209229), Point(token_index=0, time_index=1140, prob=0.9963963627815247), Point(token_index=0, time_index=1141, prob=0.9995821118354797), Point(token_index=0, time_index=1142, prob=0.999407172203064), Point(token_index=0, time_index=1143, prob=0.99859219789505), Point(token_index=0, time_index=1144, prob=0.9961361289024353), Point(token_index=0, time_index=1145, prob=0.5588687658309937), Point(token_index=1, time_index=1146, prob=0.979438304901123), Point(token_index=1, time_index=1147, prob=0.9999059438705444), Point(token_index=2, time_index=1148, prob=0.997116208076477), Point(token_index=2, time_index=1149, prob=0.9994359612464905), Point(token_index=2, time_index=1150, prob=0.9999556541442871), Point(token_index=2, time_index=1151, prob=0.3408110737800598), Point(token_index=3, time_index=1152, prob=0.989618718624115), Point(token_index=4, time_index=1153, prob=0.7447339296340942), Point(token_index=4, time_index=1154, prob=0.6670995950698853), Point(token_index=4, time_index=1155, prob=0.9998600482940674), Point(token_index=4, time_index=1156, prob=0.990929901599884), Point(token_index=4, time_index=1157, prob=0.9894214868545532), Point(token_index=4, time_index=1158, prob=0.9987514019012451), Point(token_index=5, time_index=1159, prob=0.9998664855957031), Point(token_index=5, time_index=1160, prob=0.999091386795044), Point(token_index=5, time_index=1161, prob=0.9999831914901733), Point(token_index=5, time_index=1162, prob=0.9751810431480408), Point(token_index=6, time_index=1163, prob=0.997595489025116), Point(token_index=6, time_index=1164, prob=0.9999703168869019), Point(token_index=7, time_index=1165, prob=0.9906569719314575), Point(token_index=7, time_index=1166, prob=0.9999791383743286), Point(token_index=7, time_index=1167, prob=0.999983549118042), Point(token_index=7, time_index=1168, prob=0.9999833106994629), Point(token_index=8, time_index=1169, prob=0.997403085231781), Point(token_index=8, time_index=1170, prob=0.9999816417694092), Point(token_index=9, time_index=1171, prob=0.9987607002258301), Point(token_index=9, time_index=1172, prob=0.9996192455291748), Point(token_index=9, time_index=1173, prob=0.9988454580307007), Point(token_index=9, time_index=1174, prob=0.0005588383646681905), Point(token_index=10, time_index=1175, prob=0.9996165037155151), Point(token_index=10, time_index=1176, prob=0.9995201826095581), Point(token_index=11, time_index=1177, prob=0.9992602467536926), Point(token_index=11, time_index=1178, prob=0.9995985627174377), Point(token_index=11, time_index=1179, prob=0.9999332427978516), Point(token_index=11, time_index=1180, prob=0.9996250867843628), Point(token_index=12, time_index=1181, prob=0.9990606904029846), Point(token_index=12, time_index=1182, prob=0.9997887015342712), Point(token_index=12, time_index=1183, prob=0.9999481439590454), Point(token_index=13, time_index=1184, prob=0.9998838901519775), Point(token_index=13, time_index=1185, prob=0.4883236587047577), Point(token_index=14, time_index=1186, prob=0.9987049102783203), Point(token_index=15, time_index=1187, prob=0.9875154495239258), Point(token_index=15, time_index=1188, prob=0.10120848566293716), Point(token_index=15, time_index=1189, prob=0.9988711476325989), Point(token_index=16, time_index=1190, prob=0.9992558360099792), Point(token_index=16, time_index=1191, prob=0.9998962879180908), Point(token_index=16, time_index=1192, prob=0.9999439716339111), Point(token_index=16, time_index=1193, prob=0.9999592304229736), Point(token_index=16, time_index=1194, prob=0.990139365196228), Point(token_index=17, time_index=1195, prob=0.9992726445198059), Point(token_index=17, time_index=1196, prob=0.4302506744861603), Point(token_index=17, time_index=1197, prob=0.9841821789741516), Point(token_index=18, time_index=1198, prob=0.992885410785675), Point(token_index=19, time_index=1199, prob=0.8441746830940247), Point(token_index=19, time_index=1200, prob=0.4815228581428528), Point(token_index=19, time_index=1201, prob=0.9993516802787781), Point(token_index=20, time_index=1202, prob=0.981346845626831), Point(token_index=20, time_index=1203, prob=0.9998488426208496), Point(token_index=20, time_index=1204, prob=0.9998953342437744), Point(token_index=20, time_index=1205, prob=0.9998507499694824), Point(token_index=20, time_index=1206, prob=0.9998738765716553), Point(token_index=20, time_index=1207, prob=0.9999469518661499), Point(token_index=20, time_index=1208, prob=0.9999761581420898), Point(token_index=20, time_index=1209, prob=0.9999871253967285), Point(token_index=20, time_index=1210, prob=0.999982476234436), Point(token_index=21, time_index=1211, prob=0.9752833247184753), Point(token_index=21, time_index=1212, prob=0.17559798061847687), Point(token_index=21, time_index=1213, prob=0.999976396560669), Point(token_index=21, time_index=1214, prob=0.9999759197235107), Point(token_index=21, time_index=1215, prob=0.9999734163284302), Point(token_index=21, time_index=Line truncated
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"path = Aligner.backtrack(trellis, emission, tokens)\n",
|
||||
"print(path)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Convert char path into words with timestamp"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"[<00:22.78> I'M, <00:23.05> ASLEEP, <00:23.53> AT, <00:23.71> THE, <00:23.96> WHEEL, <00:24.48> AND, <00:24.62> THERE'S, <00:24.94> A, <00:25.08> CURVE, <00:25.42> IN, <00:25.56> THE, <00:25.82> HIGHWAY, <00:30.10> YET, <00:30.33> WE, <00:30.58> TEAR, <00:30.82> THROUGH, <00:31.10> THE, <00:31.33> NIGHT, <00:31.96> LIKE, <00:32.24> A, <00:32.39> PEYOTE, <00:33.14> MIGRAINE, <00:37.06> HOW, <00:38.00> MUCH, <00:38.85> FURTHER, <00:39.92> IS, <00:40.17> IT, <00:40.89> ACHING, <00:42.61> SO, <00:43.24> EXQUISITE, <00:45.10> I, <00:45.39> HOPE, <00:46.24> WE, <00:46.70> CAN, <00:47.14> STOP, <00:47.53> IN, <00:47.89> TIME, <00:52.30> NOW, <00:52.53> YOU, <00:52.75> DON'T, <00:53.03> WANT, <00:53.25> TO, <00:53.46> DANCE, <00:54.30> 'CAUSE, <00:54.43> WE'RE, <00:54.57> LOOKING, <00:55.02> RIGHT, <00:55.30> THROUGH, <00:55.82> YOU, <00:57.92> OH, <00:58.57> OH, <00:59.10> AND, <00:59.80> THE, <01:00.07> CLOTHES, <01:00.50> THAT, <01:00.84> YOU, <01:01.07> WEAR, <01:01.74> WE, <01:01.98> DON'T, <01:02.34> MEAN, <01:02.74> TO, <01:03.35> STARE, <01:14.42> NOW, <01:14.64> THE, <01:14.90> SKY, <01:15.20> IS, <01:15.40> ON, <01:15.62> FIRE, <01:16.34> AND, <01:16.50> THE, <01:16.70> DESERT, <01:17.26> IS, <01:17.42> FREEZING, <01:21.76> WONDER, <01:22.23> HOW, <01:22.56> I'M, <01:22.82> ALIVE, <01:23.64> HEN, <01:23.85> I'M, <01:24.10> NOT, <01:24.42> EVEN, <01:24.73> BREATHING, <01:28.68> HOW, <01:29.65> MUCH, <01:30.48> FURTHER, <01:31.54> IS, <01:31.79> IT, <01:32.54> ACHING, <01:34.21> SO, <01:34.84> EXQUISITE, <01:36.62> I, <01:37.76> HOPE, <01:37.92> WE, <01:38.35> CAN, <01:38.79> STOP, <01:39.20> IN, <01:39.51> TIME, <01:43.92> NOW, <01:44.18> YOU, <01:44.40> DON'T, <01:44.65> WANT, <01:45.04> TO, <01:45.14> DANCE, <01:45.90> 'CAUSE, <01:46.04> WE'RE, <01:46.20> LOOKING, <01:46.62> RIGHT, <01:46.90> THROUGH, <01:47.29> YOU, <01:49.56> OH, <01:50.71> OH, <01:51.35> AND, <01:51.51> THE, <01:51.71> CLOTHES, <01:52.12> THAT, <01:52.42> YOU, <01:52.68> WEAR, <01:53.32> WE, <01:53.57> DON'T, <01:53.90> MEAN, <01:54.29> TO, <01:54.85> STARE, <01:56.92> OH, <01:58.04> OH, <01:58.65> BUT, <01:58.87> THE, <01:59.10> PEOPLE, <01:59.60> YOU, <01:59.85> KNOW, <02:00.68> THEY, <02:00.92> SAY, <02:01.18> WHAT, <02:01.40> THEY, <02:01.70> WANT, <02:02.15> TO, <02:04.28> OH, <02:05.46> OH, <02:05.98> 'CAUSE, <02:06.29> THE, <02:06.46> TRUTH, <02:06.84> OF, <02:07.07> IT, <02:07.29> IS, <02:08.06> WE, <02:08.30> DON'T, <02:08.68> REALLY, <02:09.59> CARE]\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from lsync.util import get_audio_segments, read_text\n",
|
||||
"from lsync.lyrics_processor import LyricsProcessor\n",
|
||||
"\n",
|
||||
"lp = LyricsProcessor(lang=lang)\n",
|
||||
"processed_lyrics = lp.process(lyrics)\n",
|
||||
"words = lp.get_words_from_path(processed_lyrics, path, frame_duration)\n",
|
||||
"\n",
|
||||
"print(words)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 4. LRC format generation"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"[00:00.00] <00:22.78> I’m <00:23.05> asleep <00:23.53> at <00:23.71> the <00:23.96> wheel <00:24.48> and <00:24.62> there’s <00:24.94> a <00:25.08> curve <00:25.42> in <00:25.56> the <00:25.82> highway\n",
|
||||
"[00:29.94] <00:30.10> Yet <00:30.33> we <00:30.58> tear <00:30.82> through <00:31.10> the <00:31.33> night <00:31.96> like <00:32.24> a <00:32.39> peyote <00:33.14> migraine\n",
|
||||
"[00:37.00] <00:37.06> How <00:38.00> much <00:38.85> further <00:39.92> is <00:40.17> it <00:40.89> aching <00:42.61> so <00:43.24> exquisite\n",
|
||||
"[00:44.92] <00:45.10> I <00:45.39> hope <00:46.24> we <00:46.70> can <00:47.14> stop <00:47.53> in <00:47.89> time\n",
|
||||
"[00:52.21] <00:52.30> Now <00:52.53> you <00:52.75> don’t <00:53.03> want <00:53.25> to <00:53.46> dance <00:54.30> 'cause <00:54.43> we’re <00:54.57> looking <00:55.02> right <00:55.30> through <00:55.82> you\n",
|
||||
"[00:57.78] <00:57.92> Oh <00:58.57> oh <00:59.10> and <00:59.80> the <01:00.07> clothes <01:00.50> that <01:00.84> you <01:01.07> wear <01:01.74> we <01:01.98> don’t <01:02.34> mean <01:02.74> to <01:03.35> stare\n",
|
||||
"[01:14.35] <01:14.42> Now <01:14.64> the <01:14.90> sky <01:15.20> is <01:15.40> on <01:15.62> fire <01:16.34> and <01:16.50> the <01:16.70> desert <01:17.26> is <01:17.42> freezing\n",
|
||||
"[01:21.70] <01:21.76> Wonder <01:22.23> how <01:22.56> I’m <01:22.82> alive <01:23.64> hen <01:23.85> I’m <01:24.10> not <01:24.42> even <01:24.73> breathing\n",
|
||||
"[01:28.62] <01:28.68> How <01:29.65> much <01:30.48> further <01:31.54> is <01:31.79> it <01:32.54> aching <01:34.21> so <01:34.84> exquisite\n",
|
||||
"[01:36.46] <01:36.62> I <01:37.76> hope <01:37.92> we <01:38.35> can <01:38.79> stop <01:39.20> in <01:39.51> time\n",
|
||||
"[01:43.84] <01:43.92> Now <01:44.18> you <01:44.40> don’t <01:44.65> want <01:45.04> to <01:45.14> dance <01:45.90> 'cause <01:46.04> we’re <01:46.20> looking <01:46.62> right <01:46.90> through <01:47.29> you\n",
|
||||
"[01:49.42] <01:49.56> Oh <01:50.71> oh <01:51.35> and <01:51.51> the <01:51.71> clothes <01:52.12> that <01:52.42> you <01:52.68> wear <01:53.32> we <01:53.57> don’t <01:53.90> mean <01:54.29> to <01:54.85> stare\n",
|
||||
"[01:56.76] <01:56.92> Oh <01:58.04> oh <01:58.65> but <01:58.87> the <01:59.10> people <01:59.60> you <01:59.85> know <02:00.68> they <02:00.92> say <02:01.18> what <02:01.40> they <02:01.70> want <02:02.15> to\n",
|
||||
"[02:04.14] <02:04.28> Oh <02:05.46> oh <02:05.98> 'cause <02:06.29> the <02:06.46> truth <02:06.84> of <02:07.07> it <02:07.29> is <02:08.06> we <02:08.30> don’t <02:08.68> really <02:09.59> care\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from lsync.lrc_formatter import LrcFormatter\n",
|
||||
"\n",
|
||||
"original_lyrics = read_text(lyrics)\n",
|
||||
"lrc = LrcFormatter.words2lrc(words, original_lyrics)\n",
|
||||
"print(lrc)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from lsync.util import save_lrc, save_words\n",
|
||||
"\n",
|
||||
"save_lrc(lrc, audio_name)\n",
|
||||
"save_words(words, audio_name)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 5. Evaluation"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Dataset (Ground Truth)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 16,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"ground_truth_path = f\"dataset/jamendolyrics/annotations/words/{audio_name}.csv\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 17,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"abs err: 0.0881\n",
|
||||
"percentage of correct segments: 0.9660\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from lsync.eval import Eval\n",
|
||||
"\n",
|
||||
"abs_err, perc_corr_segs = Eval.evaluate(words, ground_truth_path)\n",
|
||||
"print(f\"abs err: {abs_err:.4f}\\npercentage of correct segments: {perc_corr_segs:.4f}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Batch Evaluation"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 18,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<div>\n",
|
||||
"<style scoped>\n",
|
||||
" .dataframe tbody tr th:only-of-type {\n",
|
||||
" vertical-align: middle;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe tbody tr th {\n",
|
||||
" vertical-align: top;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe thead th {\n",
|
||||
" text-align: right;\n",
|
||||
" }\n",
|
||||
"</style>\n",
|
||||
"<table border=\"1\" class=\"dataframe\">\n",
|
||||
" <thead>\n",
|
||||
" <tr style=\"text-align: right;\">\n",
|
||||
" <th></th>\n",
|
||||
" <th>name</th>\n",
|
||||
" <th>audio</th>\n",
|
||||
" <th>lyrics</th>\n",
|
||||
" <th>annotation</th>\n",
|
||||
" </tr>\n",
|
||||
" </thead>\n",
|
||||
" <tbody>\n",
|
||||
" <tr>\n",
|
||||
" <th>0</th>\n",
|
||||
" <td>LUNABLIND_-_Vision__Radio_Edit_</td>\n",
|
||||
" <td>dataset/jamendolyrics/mp3/LUNABLIND_-_Vision__...</td>\n",
|
||||
" <td>dataset/jamendolyrics/lyrics/LUNABLIND_-_Visio...</td>\n",
|
||||
" <td>dataset/jamendolyrics/annotations/words/LUNABL...</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>1</th>\n",
|
||||
" <td>JASON_MILLER_-_CROWD_PLEASER</td>\n",
|
||||
" <td>dataset/jamendolyrics/mp3/JASON_MILLER_-_CROWD...</td>\n",
|
||||
" <td>dataset/jamendolyrics/lyrics/JASON_MILLER_-_CR...</td>\n",
|
||||
" <td>dataset/jamendolyrics/annotations/words/JASON_...</td>\n",
|
||||
" </tr>\n",
|
||||
" </tbody>\n",
|
||||
"</table>\n",
|
||||
"</div>"
|
||||
],
|
||||
"text/plain": [
|
||||
" name \\\n",
|
||||
"0 LUNABLIND_-_Vision__Radio_Edit_ \n",
|
||||
"1 JASON_MILLER_-_CROWD_PLEASER \n",
|
||||
"\n",
|
||||
" audio \\\n",
|
||||
"0 dataset/jamendolyrics/mp3/LUNABLIND_-_Vision__... \n",
|
||||
"1 dataset/jamendolyrics/mp3/JASON_MILLER_-_CROWD... \n",
|
||||
"\n",
|
||||
" lyrics \\\n",
|
||||
"0 dataset/jamendolyrics/lyrics/LUNABLIND_-_Visio... \n",
|
||||
"1 dataset/jamendolyrics/lyrics/JASON_MILLER_-_CR... \n",
|
||||
"\n",
|
||||
" annotation \n",
|
||||
"0 dataset/jamendolyrics/annotations/words/LUNABL... \n",
|
||||
"1 dataset/jamendolyrics/annotations/words/JASON_... "
|
||||
]
|
||||
},
|
||||
"execution_count": 18,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from lsync.eval import get_eval_df\n",
|
||||
"\n",
|
||||
"eval_df = get_eval_df()\n",
|
||||
"eval_df.head(2)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 19,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Some weights of Wav2Vec2ForCTC were not initialized from the model checkpoint at facebook/wav2vec2-large-960h-lv60-self and are newly initialized: ['wav2vec2.masked_spec_embed']\n",
|
||||
"You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
|
||||
"20it [07:27, 22.36s/it]\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"<div>\n",
|
||||
"<style scoped>\n",
|
||||
" .dataframe tbody tr th:only-of-type {\n",
|
||||
" vertical-align: middle;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe tbody tr th {\n",
|
||||
" vertical-align: top;\n",
|
||||
" }\n",
|
||||
"\n",
|
||||
" .dataframe thead th {\n",
|
||||
" text-align: right;\n",
|
||||
" }\n",
|
||||
"</style>\n",
|
||||
"<table border=\"1\" class=\"dataframe\">\n",
|
||||
" <thead>\n",
|
||||
" <tr style=\"text-align: right;\">\n",
|
||||
" <th></th>\n",
|
||||
" <th>name</th>\n",
|
||||
" <th>average_abs_err</th>\n",
|
||||
" <th>percentage_of_correct_segments</th>\n",
|
||||
" </tr>\n",
|
||||
" </thead>\n",
|
||||
" <tbody>\n",
|
||||
" <tr>\n",
|
||||
" <th>0</th>\n",
|
||||
" <td>Lower_Loveday_-_Is_It_Right_</td>\n",
|
||||
" <td>0.240656</td>\n",
|
||||
" <td>0.971698</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>1</th>\n",
|
||||
" <td>Ridgway_-_Fire_Inside</td>\n",
|
||||
" <td>1.025926</td>\n",
|
||||
" <td>0.842105</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>2</th>\n",
|
||||
" <td>Wordsmith_-_The_Statement</td>\n",
|
||||
" <td>0.091202</td>\n",
|
||||
" <td>0.979346</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>3</th>\n",
|
||||
" <td>The.madpix.project_-_One_Way_Street</td>\n",
|
||||
" <td>1.905688</td>\n",
|
||||
" <td>0.896175</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>4</th>\n",
|
||||
" <td>Moon_I_Mean_-_Wrong_Concept</td>\n",
|
||||
" <td>0.121865</td>\n",
|
||||
" <td>0.973783</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>5</th>\n",
|
||||
" <td>Kinematic_-_Peyote</td>\n",
|
||||
" <td>0.086068</td>\n",
|
||||
" <td>0.972789</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>6</th>\n",
|
||||
" <td>Quentin_Hannappe_-_Keep_On</td>\n",
|
||||
" <td>0.114757</td>\n",
|
||||
" <td>0.982857</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>7</th>\n",
|
||||
" <td>Explosive_Ear_Candy_-_Like_The_Sun</td>\n",
|
||||
" <td>0.113410</td>\n",
|
||||
" <td>0.992453</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>8</th>\n",
|
||||
" <td>Pure_Mids_-_The_Leader</td>\n",
|
||||
" <td>3.736285</td>\n",
|
||||
" <td>0.701754</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>9</th>\n",
|
||||
" <td>HILA_-_Give_Me_the_Same</td>\n",
|
||||
" <td>0.213927</td>\n",
|
||||
" <td>0.919255</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>10</th>\n",
|
||||
" <td>Cortez_-_Feel__Stripped_</td>\n",
|
||||
" <td>0.167805</td>\n",
|
||||
" <td>0.963173</td>\n",
|
||||
" </tr>\n",
|
||||
" <tr>\n",
|
||||
" <th>11</th>\n",
|
||||
" <td>Rxbyn_-_Bad_Side</td>\n",
|
||||
" <td>0.137684</td>\n",
|
||||
" <td>0.956818</td>\n",
|
||||
" </tr>\n",
|
||||
" </tbody>\n",
|
||||
"</table>\n",
|
||||
"</div>"
|
||||
],
|
||||
"text/plain": [
|
||||
" name average_abs_err \\\n",
|
||||
"0 Lower_Loveday_-_Is_It_Right_ 0.240656 \n",
|
||||
"1 Ridgway_-_Fire_Inside 1.025926 \n",
|
||||
"2 Wordsmith_-_The_Statement 0.091202 \n",
|
||||
"3 The.madpix.project_-_One_Way_Street 1.905688 \n",
|
||||
"4 Moon_I_Mean_-_Wrong_Concept 0.121865 \n",
|
||||
"5 Kinematic_-_Peyote 0.086068 \n",
|
||||
"6 Quentin_Hannappe_-_Keep_On 0.114757 \n",
|
||||
"7 Explosive_Ear_Candy_-_Like_The_Sun 0.113410 \n",
|
||||
"8 Pure_Mids_-_The_Leader 3.736285 \n",
|
||||
"9 HILA_-_Give_Me_the_Same 0.213927 \n",
|
||||
"10 Cortez_-_Feel__Stripped_ 0.167805 \n",
|
||||
"11 Rxbyn_-_Bad_Side 0.137684 \n",
|
||||
"\n",
|
||||
" percentage_of_correct_segments \n",
|
||||
"0 0.971698 \n",
|
||||
"1 0.842105 \n",
|
||||
"2 0.979346 \n",
|
||||
"3 0.896175 \n",
|
||||
"4 0.973783 \n",
|
||||
"5 0.972789 \n",
|
||||
"6 0.982857 \n",
|
||||
"7 0.992453 \n",
|
||||
"8 0.701754 \n",
|
||||
"9 0.919255 \n",
|
||||
"10 0.963173 \n",
|
||||
"11 0.956818 "
|
||||
]
|
||||
},
|
||||
"execution_count": 19,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from lsync import LyricsSync\n",
|
||||
"\n",
|
||||
"lsync = LyricsSync(lang=lang, blank_id=blank_id)\n",
|
||||
"Eval.eval_all(eval_df, lsync)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Result of finetuned model in singing domain based on facebook/wav2vec2-base"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 20,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Avg abs err: 0.6246593301330764 | Avg percentage: 0.9292648064655545\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import pandas as pd\n",
|
||||
"eval_result = pd.read_csv('eval_finetune.csv')\n",
|
||||
"print(\"Avg abs err: {} | Avg percentage: {}\".format(\n",
|
||||
" eval_result.loc[:, 'average_abs_err'].mean(),\n",
|
||||
" eval_result.loc[:, 'percentage_of_correct_segments'].mean(),\n",
|
||||
"))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Result of large model of facebook/wav2vec2-large-960h-lv60-self in speech domain"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 21,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Avg abs err: 0.8945763391872722 | Avg percentage: 0.9131257208477407\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import pandas as pd\n",
|
||||
"eval_result = pd.read_csv('eval_large.csv')\n",
|
||||
"print(\"Avg abs err: {} | Avg percentage: {}\".format(\n",
|
||||
" eval_result.loc[:, 'average_abs_err'].mean(),\n",
|
||||
" eval_result.loc[:, 'percentage_of_correct_segments'].mean(),\n",
|
||||
"))"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "lsync",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.7.12"
|
||||
},
|
||||
"orig_nbformat": 4
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
Executable
+229
@@ -0,0 +1,229 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from datasets import Dataset\n",
|
||||
"import numpy as np\n",
|
||||
"import torch\n",
|
||||
"from lsync.config import TARGET_SR\n",
|
||||
"import librosa\n",
|
||||
"from transformers import Wav2Vec2Processor, Wav2Vec2FeatureExtractor, Wav2Vec2CTCTokenizer\n",
|
||||
"\n",
|
||||
"import evaluate\n",
|
||||
"\n",
|
||||
"metrics = evaluate.load(\"wer\")\n",
|
||||
"MODEL_ID = \"facebook/wav2vec2-base\"\n",
|
||||
"\n",
|
||||
"tokenizer = Wav2Vec2CTCTokenizer(\"vocab.json\", unk_token=\"[UNK]\", pad_token=\"[PAD]\", word_delimiter_token=\"|\")\n",
|
||||
"feature_extractor = Wav2Vec2FeatureExtractor(feature_size=1, sampling_rate=16000, padding_value=0.0, do_normalize=True, return_attention_mask=False)\n",
|
||||
"processor = Wav2Vec2Processor(feature_extractor=feature_extractor, tokenizer=tokenizer)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Dataset"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def audio_processing(fp):\n",
|
||||
" y, sr = librosa.load(fp, sr=TARGET_SR)\n",
|
||||
" yt, _ = librosa.effects.trim(y, top_db=30)\n",
|
||||
" return yt\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def data_preprocess(data):\n",
|
||||
" audio = audio_processing(data['path'])\n",
|
||||
" result = {}\n",
|
||||
" result['input_values'] = processor(\n",
|
||||
" audio, sampling_rate=TARGET_SR).input_values[0]\n",
|
||||
" with processor.as_target_processor():\n",
|
||||
" result[\"labels\"] = processor(data[\"text\"]).input_ids\n",
|
||||
" return result\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def prepare_dataset(batch):\n",
|
||||
" audio = batch[\"path\"]\n",
|
||||
"\n",
|
||||
" # batched output is \"un-batched\" to ensure mapping is correct\n",
|
||||
" batch[\"input_values\"] = processor(\n",
|
||||
" audio[\"array\"], sampling_rate=audio[\"sampling_rate\"]).input_values[0]\n",
|
||||
" batch[\"labels\"] = processor(batch[\"text\"]).input_ids\n",
|
||||
" return batch\n",
|
||||
"\n",
|
||||
"def get_datasets():\n",
|
||||
" dataset_csv_path = \"dataset.csv\"\n",
|
||||
" dataset = Dataset.from_csv(dataset_csv_path)\n",
|
||||
" dataset = dataset.filter(lambda x: x[\"text\"] != \"\")\n",
|
||||
" dataset.cleanup_cache_files()\n",
|
||||
" dataset = dataset.train_test_split(test_size=0.05, seed=41)\n",
|
||||
" # smaller_dataset = dataset['test'].train_test_split(test_size=0.05)\n",
|
||||
" train_dataset = dataset['train']\n",
|
||||
" test_dataset = dataset['test']\n",
|
||||
"\n",
|
||||
" train_dataset = train_dataset.map(\n",
|
||||
" data_preprocess,\n",
|
||||
" num_proc=8\n",
|
||||
" )\n",
|
||||
" test_dataset = test_dataset.map(\n",
|
||||
" data_preprocess,\n",
|
||||
" num_proc=8\n",
|
||||
" )\n",
|
||||
" return (train_dataset, test_dataset)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"train_dataset, test_dataset = get_datasets()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Train"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"@dataclass\n",
|
||||
"class DataCollatorCTCWithPadding:\n",
|
||||
" processor: Wav2Vec2Processor\n",
|
||||
" padding: Union[bool, str] = True\n",
|
||||
" max_length: Optional[int] = None\n",
|
||||
" max_length_labels: Optional[int] = None\n",
|
||||
" pad_to_multiple_of: Optional[int] = None\n",
|
||||
" pad_to_multiple_of_labels: Optional[int] = None\n",
|
||||
"\n",
|
||||
" def __call__(self, features: List[Dict[str, Union[List[int], torch.Tensor]]]) -> Dict[str, torch.Tensor]:\n",
|
||||
" input_features = [{\"input_values\": feature[\"input_values\"]}\n",
|
||||
" for feature in features]\n",
|
||||
" label_features = [{\"input_ids\": feature[\"labels\"]}\n",
|
||||
" for feature in features]\n",
|
||||
"\n",
|
||||
" batch = self.processor.pad(\n",
|
||||
" input_features,\n",
|
||||
" padding=self.padding,\n",
|
||||
" max_length=self.max_length,\n",
|
||||
" pad_to_multiple_of=self.pad_to_multiple_of,\n",
|
||||
" return_tensors=\"pt\",\n",
|
||||
" )\n",
|
||||
" with self.processor.as_target_processor():\n",
|
||||
" labels_batch = self.processor.pad(\n",
|
||||
" label_features,\n",
|
||||
" padding=self.padding,\n",
|
||||
" max_length=self.max_length_labels,\n",
|
||||
" pad_to_multiple_of=self.pad_to_multiple_of_labels,\n",
|
||||
" return_tensors=\"pt\",\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" labels = labels_batch[\"input_ids\"].masked_fill(\n",
|
||||
" labels_batch.attention_mask.ne(1), -100)\n",
|
||||
"\n",
|
||||
" batch[\"labels\"] = labels\n",
|
||||
"\n",
|
||||
" return batch\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def compute_metrics(pred):\n",
|
||||
" pred_logits = pred.predictions\n",
|
||||
" pred_ids = np.argmax(pred_logits, axis=-1)\n",
|
||||
" pred.label_ids[pred.label_ids == -100] = processor.tokenizer.pad_token_id\n",
|
||||
" pred_str = processor.batch_decode(pred_ids)\n",
|
||||
" label_str = processor.batch_decode(pred.label_ids, group_tokens=False)\n",
|
||||
" wer = metrics.compute(predictions=pred_str, references=label_str)\n",
|
||||
" return {\"wer\": wer}"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Train\n",
|
||||
"data_collator = DataCollatorCTCWithPadding(\n",
|
||||
" processor=processor, padding=True)\n",
|
||||
"\n",
|
||||
"model = Wav2Vec2ForCTC.from_pretrained(\n",
|
||||
" MODEL_ID,\n",
|
||||
" ctc_loss_reduction=\"mean\",\n",
|
||||
" pad_token_id=processor.tokenizer.pad_token_id,\n",
|
||||
")\n",
|
||||
"model.freeze_feature_extractor()\n",
|
||||
"torch.cuda.empty_cache()\n",
|
||||
"training_args = TrainingArguments(\n",
|
||||
" output_dir=\"model\",\n",
|
||||
" per_device_train_batch_size=2,\n",
|
||||
" per_device_eval_batch_size=2,\n",
|
||||
" gradient_accumulation_steps=2,\n",
|
||||
" num_train_epochs=5,\n",
|
||||
" fp16=True,\n",
|
||||
" save_steps=3000,\n",
|
||||
" eval_steps=20000,\n",
|
||||
" logging_steps=2000,\n",
|
||||
" learning_rate=5e-4,\n",
|
||||
" weight_decay=0.0001,\n",
|
||||
" warmup_steps=1000,\n",
|
||||
" save_total_limit=8,\n",
|
||||
" evaluation_strategy=\"steps\",\n",
|
||||
" # load_best_model_at_end=True,\n",
|
||||
" # metric_for_best_model=\"eval_wer\"\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"trainer = Trainer(\n",
|
||||
" model=model,\n",
|
||||
" data_collator=data_collator,\n",
|
||||
" args=training_args,\n",
|
||||
" train_dataset=train_dataset,\n",
|
||||
" eval_dataset=test_dataset,\n",
|
||||
" tokenizer=processor.feature_extractor,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"trainer.train(resume_from_checkpoint=True)"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "lsync",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.7.12"
|
||||
},
|
||||
"orig_nbformat": 4
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
Executable
+1
@@ -0,0 +1 @@
|
||||
{"'": 1, "a": 2, "b": 3, "c": 4, "d": 5, "e": 6, "f": 7, "g": 8, "h": 9, "i": 10, "j": 11, "k": 12, "l": 13, "m": 14, "n": 15, "o": 16, "p": 17, "q": 18, "r": 19, "s": 20, "t": 21, "u": 22, "v": 23, "w": 24, "x": 25, "y": 26, "z": 27, "|": 0, "[UNK]": 28, "[PAD]": 29}
|
||||
Reference in new issue
Block a user