File size: 7,944 Bytes
82b1107 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 |
---
license: cc-by-nc-4.0
---
# Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement
[](https://openreview.net/pdf?id=anQDiQZhDP)
[](https://huggingface.co/amphion/Vevo)
[](https://versavoice.github.io/) [](https://github.com/open-mmlab/Amphion/blob/main/models/vc/vevo/README.md)
We present our reproduction of [Vevo](https://openreview.net/pdf?id=anQDiQZhDP), a versatile zero-shot voice imitation framework with controllable timbre and style. We invite you to explore our [audio samples](https://versavoice.github.io/) at to experience Vevo's capabilities firsthand.
We have included the following pre-trained Vevo models at Amphion:
- **Vevo-Timbre**: It can conduct *style-preserved* voice conversion.
- **Vevo-Style**: It can conduct style conversion, such as *accent conversion* and *emotion conversion*.
- **Vevo-Voice**: It can conduct *style-converted* voice conversion.
- **Vevo-TTS**: It can conduct *style and timbre controllable* TTS.
Besides, we also release the **content tokenizer** and **content-style tokenizer** proposed by Vevo. Notably, all these pre-trained models are trained on [Emilia](https://huggingface.co/datasets/amphion/Emilia-Dataset), containing 101k hours of speech data among six languages (English, Chinese, German, French, Japanese, and Korean).
## Model Introduction
We provide the following pre-trained models:
| Model Name | Description |
|-------------------|-------------|
| [Content Tokenizer](https://huggingface.co/amphion/Vevo/tree/main/tokenizer/vq32) | Converting speech to content tokens. It is a single codebook VQ-VAE with a vocabulary size of 32. The frame rate is 50Hz.|
| [Content-Style Tokenizer](https://huggingface.co/amphion/Vevo/tree/main/tokenizer/vq8192) | Converting speech to content-style tokens. It is a single codebook VQ-VAE with a vocabulary size of 8192. The frame rate is 50Hz.|
| [Vq32ToVq8192](https://huggingface.co/amphion/Vevo/tree/main/contentstyle_modeling/Vq32ToVq8192) | Predicting content-style tokens from content tokens with an auto-regressive transformer (480M). |
| [PhoneToVq8192](https://huggingface.co/amphion/Vevo/tree/main/contentstyle_modeling/PhoneToVq8192) | Predicting content-style tokens from phone tokens with an auto-regressive transformer (740M). |
| [Vq8192ToMels](https://huggingface.co/amphion/Vevo/tree/main/acoustic_modeling/Vq8192ToMels) | Predicting mel-spectrogram from content-style tokens with a flow-matching transformer (330M). |
| [Vocoder](https://huggingface.co/amphion/Vevo/tree/main/acoustic_modeling/Vocoder) | Predicting audio from mel-spectrogram with a Vocos-based vocoder (250M). |
You can download all pretrained checkpoints from [HuggingFace](https://huggingface.co/amphion/MaskGCT/tree/main) or use huggingface API.
## Usage
You can refer to our [recipe](https://github.com/open-mmlab/Amphion/blob/main/models/vc/vevo/README.md) at GitHub for more usage details. For example, to use Vevo-TTS, after you clone the Amphion github repository, you can use the script like:
```python
import os
from huggingface_hub import snapshot_download
from models.vc.vevo.vevo_utils import *
def vevo_tts(
src_text,
ref_wav_path,
timbre_ref_wav_path=None,
output_path=None,
ref_text=None,
src_language="en",
ref_language="en",
):
if timbre_ref_wav_path is None:
timbre_ref_wav_path = ref_wav_path
gen_audio = inference_pipeline.inference_ar_and_fm(
src_wav_path=None,
src_text=src_text,
style_ref_wav_path=ref_wav_path,
timbre_ref_wav_path=timbre_ref_wav_path,
style_ref_wav_text=ref_text,
src_text_language=src_language,
style_ref_wav_text_language=ref_language,
)
assert output_path is not None
save_audio(gen_audio, output_path=output_path)
if __name__ == "__main__":
# ===== Device =====
device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
# ===== Content-Style Tokenizer =====
local_dir = snapshot_download(
repo_id="amphion/Vevo",
repo_type="model",
cache_dir="./ckpts/Vevo",
allow_patterns=["tokenizer/vq8192/*"],
)
content_style_tokenizer_ckpt_path = os.path.join(local_dir, "tokenizer/vq8192")
# ===== Autoregressive Transformer =====
local_dir = snapshot_download(
repo_id="amphion/Vevo",
repo_type="model",
cache_dir="./ckpts/Vevo",
allow_patterns=["contentstyle_modeling/PhoneToVq8192/*"],
)
ar_cfg_path = "./models/vc/vevo/config/PhoneToVq8192.json"
ar_ckpt_path = os.path.join(local_dir, "contentstyle_modeling/PhoneToVq8192")
# ===== Flow Matching Transformer =====
local_dir = snapshot_download(
repo_id="amphion/Vevo",
repo_type="model",
cache_dir="./ckpts/Vevo",
allow_patterns=["acoustic_modeling/Vq8192ToMels/*"],
)
fmt_cfg_path = "./models/vc/vevo/config/Vq8192ToMels.json"
fmt_ckpt_path = os.path.join(local_dir, "acoustic_modeling/Vq8192ToMels")
# ===== Vocoder =====
local_dir = snapshot_download(
repo_id="amphion/Vevo",
repo_type="model",
cache_dir="./ckpts/Vevo",
allow_patterns=["acoustic_modeling/Vocoder/*"],
)
vocoder_cfg_path = "./models/vc/vevo/config/Vocoder.json"
vocoder_ckpt_path = os.path.join(local_dir, "acoustic_modeling/Vocoder")
# ===== Inference =====
inference_pipeline = VevoInferencePipeline(
content_style_tokenizer_ckpt_path=content_style_tokenizer_ckpt_path,
ar_cfg_path=ar_cfg_path,
ar_ckpt_path=ar_ckpt_path,
fmt_cfg_path=fmt_cfg_path,
fmt_ckpt_path=fmt_ckpt_path,
vocoder_cfg_path=vocoder_cfg_path,
vocoder_ckpt_path=vocoder_ckpt_path,
device=device,
)
src_text = "I don't really care what you call me. I've been a silent spectator, watching species evolve, empires rise and fall. But always remember, I am mighty and enduring. Respect me and I'll nurture you; ignore me and you shall face the consequences."
ref_wav_path = "./models/vc/vevo/wav/arabic_male.wav"
ref_text = "Flip stood undecided, his ears strained to catch the slightest sound."
# 1. Zero-Shot TTS (the style reference and timbre reference are same)
vevo_tts(
src_text,
ref_wav_path,
output_path="./models/vc/vevo/wav/output_vevotts1.wav",
ref_text=ref_text,
src_language="en",
ref_language="en",
)
# 2. Style and Timbre Controllable Zero-Shot TTS (the style reference and timbre reference are different)
vevo_tts(
src_text,
ref_wav_path,
timbre_ref_wav_path="./models/vc/vevo/wav/mandarin_female.wav",
output_path="./models/vc/vevo/wav/output_vevotts2.wav",
ref_text=ref_text,
src_language="en",
ref_language="en",
)
```
## Citation
If you use Vevo in your research, please cite the following papers:
```bibtex
@article{vevo,
title={Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement},
journal={OpenReview},
year={2024}
}
@inproceedings{amphion,
author={Zhang, Xueyao and Xue, Liumeng and Gu, Yicheng and Wang, Yuancheng and Li, Jiaqi and He, Haorui and Wang, Chaoren and Song, Ting and Chen, Xi and Fang, Zihao and Chen, Haopeng and Zhang, Junan and Tang, Tze Ying and Zou, Lexiao and Wang, Mingxuan and Han, Jun and Chen, Kai and Li, Haizhou and Wu, Zhizheng},
title={Amphion: An Open-Source Audio, Music and Speech Generation Toolkit},
booktitle={{IEEE} Spoken Language Technology Workshop, {SLT} 2024},
year={2024}
}
``` |