Hugging Face's logo

dkounadis
/
artificial-styletts2

Text-to-Audio
Audiocraft
English
audiogen
styletts2
shift-tts
sound
audio-generation
text-to-speech
mimic3
Model card Files Community
4
artificial-styletts2 / README.md
Dionyssos's picture
Dionyssos
audiocraft x Nrepeat
d2ffdd6
preview code
|
raw
history blame
3.06 kB
metadata
license: mit
language:
  - en
pipeline_tag: text-to-speech
tags:
  - audiocraft
  - audiogen
  - styletts2
  - audio
  - synthesis
  - shift
  - audeering
  - dkounadis
  - sound
  - scene
  - acoustic-scene
  - audio-generation

Affective TTS - SoundScape

  • SHIFT TTS tool
  • Soundscapes e.g. trees, water, leaves,, generation via AudioGen
  • landscape2soundscape.py shows how to overlay TTS & Soundscape to Images and create videos
  • 134 TTS Voices - Affective / Non-affective mode for every voice

Available Voices

Listen to available voices!

Flask API

Install

virtualenv --python=python3 ~/.envs/.my_env
source ~/.envs/.my_env/bin/activate
cd shift/
pip install -r requirements.txt

Flask

CUDA_DEVICE_ORDER=PCI_BUS_ID HF_HOME=./hf_home CUDA_VISIBLE_DEVICES=2 python api.py

The following need api.py to be running on a tmux session.

Text 2 Speech 

# Basic TTS - See Available Voices
python tts.py --text sample.txt --voice "en_US/m-ailabs_low#mary_ann" --affective

# voice cloning
python tts.py --text sample.txt --native assets/native_voice.wav

Image 2 Video 

# Make video narrating an image - All above TTS args apply also here!
python tts.py --text sample.txt --image assets/image_from_T31.jpg

Video 2 Video 

# Video Dubbing - from time-stamped subtitles (.srt)
python tts.py --text assets/head_of_fortuna_en.srt --video assets/head_of_fortuna.mp4

# Video narration - from text description (.txt)
python tts.py --text assets/head_of_fortuna_GPT.txt --video assets/head_of_fortuna.mp4

Landscape 2 Soundscape 

# TTS & soundscape - overlay to .mp4
python landscape2soundscape.py

Examples

Substitute Native voice via TTS

Native voice ANBPR video

Same video where Native voice is replaced with English TTS voice with similar emotion

Same video w. Native voice replaced with English TTS

Video dubbing from subtitles .srt

Video Dubbing

Review demo SHIFT

Generate dubbed video:

python tts.py --text assets/head_of_fortuna_en.srt --video assets/head_of_fortuna.mp4

Joint Application of D3.1 & D3.2

Foo4

From an image and text create a video:


python tts.py --text sample.txt --image assets/image_from_T31.jpg

Live Demo - Paplay

Flask

CUDA_DEVICE_ORDER=PCI_BUS_ID HF_HOME=/data/dkounadis/.hf7/ CUDA_VISIBLE_DEVICES=4 python live_api.py

Client (Ubutu)

python live_demo.py  # will ask text input & play soundscape