text_to_speech
1.0.0
Check the CHANGELOG file to have a global overview of the latest modifications ! ?
├── custom_architectures
│ ├── tacotron2_arch.py : Tacotron-2 synthesizer architecture
│ └── waveglow_arch.py : WaveGlow vocoder architecture
├── custom_layers
├── custom_train_objects
│ ├── losses
│ │ └── tacotron_loss.py : custom Tacotron2 loss
├── example_outputs : some pre-computed audios (cf the `text_to_speech` notebook)
├── loggers
├── models
│ ├── encoder : the `AudioEncoder` is used as speaker encoder for the SV2TTS model*
│ ├── tts
│ │ ├── sv2tts_tacotron2.py : SV2TTS main class
│ │ ├── tacotron2.py : Tacotron2 main class
│ │ ├── vocoder.py : main functions for complete inference
│ │ └── waveglow.py : WaveGlow main class (both pytorch and tensorflow)
├── pretrained_models
├── unitests
├── utils
├── example_fine_tuning.ipynb
├── example_sv2tts.ipynb
├── example_tacotron2.ipynb
├── example_waveglow.ipynb
└── text_to_speech.ipynbCheck the main project for more information about the unextended modules / structure / main classes.
* Check the encoders project for more information about the models/encoder module
models.tts) :| Feature | Fuction / class | Description |
|---|---|---|
| Text-To-Speech | tts |
perform TTS on text you want with the model you want |
| stream | tts_stream |
perform TTS on text you enter |
| TTS logger | loggers.TTSLogger |
converts logging logs to voice and play it |
The text_to_speech notebook provides a concrete demonstration of the tts function
Available architectures :
Synthesizer :
SV2TTS)Vocoder :
The SV2TTS models are fine-tuned from pretrained Tacotron2 models, by using the partial transfer learning procedure (see below for details), which speeds up a lot the training.
| Name | Language | Dataset | Synthesizer | Vocoder | Speaker Encoder | Trainer | Weights |
|---|---|---|---|---|---|---|---|
| pretrained_tacotron2 | en |
LJSpeech | Tacotron2 |
WaveGlow |
/ | NVIDIA | Google Drive |
| tacotron2_siwis | fr |
SIWIS | Tacotron2 |
WaveGlow |
/ | me | Google Drive |
| sv2tts_tacotron2_256 | fr |
SIWIS, VoxForge, CommonVoice | SV2TTSTacotron2 |
WaveGlow |
Google Drive | me | Google Drive |
| sv2tts_siwis | fr |
SIWIS, VoxForge, CommonVoice | SV2TTSTacotron2 |
WaveGlow |
Google Drive | me | Google Drive |
| sv2tts_tacotron2_256_v2 | fr |
SIWIS, VoxForge, CommonVoice | SV2TTSTacotron2 |
WaveGlow |
Google Drive | me | Google Drive |
| sv2tts_siwis_v2 | fr |
SIWIS | SV2TTSTacotron2 |
WaveGlow |
Google Drive | me | Google Drive |
Models must be unzipped in the pretrained_models/ directory !
Important Note : the NVIDIA models available on torch hub requires a compatible GPU with the correct configuration for pytorch. It is the reason why the both models are provided in the expected keras checkpoint ?
The sv2tts_siwis models are fine-tuned version of sv2tts_tacotron2_256 on the SIWIS (single-speaker) dataset. Fine-tuning a multi-speaker on a single-speaker dataset tends to improve the stability, and to produce a voice with more intonation, compared to simply training the single-speaker model.
A Google Colab demo is available at this link !
You can also find some audio generated in example_outputs/, or directly in the Colab notebook ;)
Check this installagion guide for the step-by-step instructions !
You may have to install ffmpeg for audio loading / saving.
Google Colab demonstrationtensorflow 2.xbatch_size support for vocoder inferenceSV2TTS weightssimilarity loss to test a new training procedure for single-speaker fine-tuningTTS on document (in progress)SV2TTS model based on an encoder trained with the GE2E lossThere are multiple ways to enable multi-speaker speech synthesis :
speaker ID that is embedded by a learnable Embedding layer. The speaker embedding is then learned during training.Speaker Encoder (SE) to embed audio from the reference speaker. This is often referred as zero-shot voice cloning, as it only requires a sample from the speaker (without training).prompt-based strategy has been proposed to control the speech with prompts.SV2TTS architectureNote : in the next paragraphs, encoder refers to the Tacotron Encoder part, while SE refers to a speaker encoder model (detailed below).
The Speaker Encoder-based Text-To-Speech is inspired from the "From Speaker Verification To Text-To-Speech (SV2TTS)" paper. The authors have proposed an extension of the Tacotron-2 architecture to include information about the speaker's voice.
Here is a short overview of the proposed procedure :
speaker verification model. This model basically takes as input an audio sample (5-10 sec) from a speaker, and encodes it on a d-dimensional vector, named the embedding. This embedding aims to capture relevant information about the speaker's voice (e.g., frequencies, rythm, pitch, ...).Speaker Encoder (SE) is then used to encode the voice of the speaker to clone.Tacotron-2 encoder part, such that the Decoder has access to both the encoded text and the speaker embedding.The objective is that the Decoder will learn to use the speaker embedding to copy its prosody / intonation / ... to read the text with the voice of this speaker.
There are some limitations with the above approach :
Speaker Encoder has to correctly separate speakers, and encode their voice in a meaningful way for the synthesizer.To tackle these limitations, the proposed solution is to perform a 2-step training :
CommonVoice database. This is one of the largest multilingual database for audio, at the cost of noisy / variable quality audios. This is therefore not suitable to train good quality models, whereas pre-processing still helps to obtain intelligible audios.The SE part should be able to differentiate speakers, and embed (encode in a 1-D vector) them in a meaningful way.
The model used in the paper is a 3-layer LSTM model with a normalization layer trained with the GE2E loss. The major limitation is that training this model is really slow, and took 2 weeks on 4 GPU's in the CorentinJ master thesis (cf his github)
This project proposes a simpler architecture based on Convolutional Neural Networks (CNN), which is much faster to train compared to LSTM networks. Furthermore, the euclidian distance has been used rather than the cosine metric, which has shown faster convergence. Additionally, a custom cache-based generator is proposed to speed up audio processing. These modifications allowed to train a 99% accuracy model within 2-3 hours on a single RTX 3090 GPU !
In order to avoid training a SV2TTS model from scratch, which would be completely impossible on a single GPU, a new partial transfer learning procedure is proposed.
This procedure takes a pre-trained model with a slightly different architecture, and transfer all the common weights (like in regular transfer learning). For the layers with different weights shape, only the common part is transfered, while the remaining weights are initialized to zeros. This result in a new model with different weights to mimic the behavior of the original model.
In the SV2TTS architecture, the speaker embedding is passed to the recurrent layer of the Tacotron2 decoder. This results in a different input shape, making the layer weights matrix different. The partial transfer learning allows to nitialize the model such that it replicates the behavior of the original single-speaker Tacotron2 model !
Contacts :
[email protected]The goal of these projects is to support and advance education and research in Deep Learning technology. To facilitate this, all associated code is made available under the GNU Affero General Public License (AGPL) v3, supplemented by a clause that prohibits commercial use (cf the LICENCE file).
These projects are released as "free software", allowing you to freely use, modify, deploy, and share the software, provided you adhere to the terms of the license. While the software is freely available, it is not public domain and retains copyright protection. The license conditions are designed to ensure that every user can utilize and modify any version of the code for their own educational and research projects.
If you wish to use this project in a proprietary commercial endeavor, you must obtain a separate license. For further details on this process, please contact me directly.
For my protection, it is important to note that all projects are available on an "As Is" basis, without any warranties or conditions of any kind, either explicit or implied. However, do not hesitate to report issues on the repository's project, or make a Pull Request to solve it ?
If you find this project useful in your work, please add this citation to give it more visibility ! ?
@misc{yui-mhcp
author = {yui},
title = {A Deep Learning projects centralization},
year = {2021},
publisher = {GitHub},
howpublished = {url{https://github.com/yui-mhcp}}
}
The code for this project is a mixture of multiple GitHub projects, to have a fully modulable Tacotron-2 implementation
dynamic decoder implementation, which has now been optimized and updated to be Keras 3 compatible.SV2TTS architecture is inspired from this repository, with small differences and optimizations.Papers :
