fx non-periodicity of sound background

README.md

@@ -21,6 +21,7 @@ tags:
 
 # Affective TTS - SoundScape
   - [SHIFT TTS tool](https://github.com/audeering/shift)
+  - Analysis of Emotionality [Issue #1](https://huggingface.co/dkounadis/artificial-styletts2/discussions/2)
   - Soundscapes `e.g. trees, water, leaves,`, generation via [AudioGen](https://huggingface.co/dkounadis/artificial-styletts2/discussions/3)
   - `landscape2soundscape.py` shows how to overlay TTS & Soundscape to Images and create videos
   - `134` TTS Voices - Affective / Non-affective mode for every voice
@@ -130,6 +131,17 @@ From an image and text create a video:
 python tts.py --text sample.txt --image assets/image_from_T31.jpg
 ```
 
+## Landscape 2 Soundscape
+
+<a href="https://www.youtube.com/watch?v=FCYS29OrA1s" rel='01_Schick_AII840_001.jpg'>![01_Schick_AII840_001](uc_spk_Landscape2Soundscape_Masterpieces_pics/01_Schick_AII840_001.jpg)
+
+
+
+```python
+# Loads image & text & sound-scene text and creates .mp4
+python landscape2soundscape.py
+```
+
 
 # Live Demo - Paplay
 

api.py

@@ -45,24 +45,40 @@ def _shift(x):
 def overlay(x, scene=None):
     if scene is not None:
         
-        n_repeat = len(x) // ((SOUNDSCAPE_DURATION + 1) * 24000)
-        sound_background = []
-        for _ in range(n_repeat):
-            print(f'AudioCraft: {_} of {n_repeat}', end='\r')
-            x = sound_generator.generate(
-                                            [scene]
-                                                )[0].detach().cpu().numpy()[0, :]
-            x = audresample.resample(x, 
-                                            original_rate=sound_generator.sample_rate,  # 16000
-                                            target_rate=24000)[0, :]
-            sound_background.append(x)
-        sound_background = np.concatenate(sound_background)
-
-        for d in range(10):
-                # roll long audio to concentrate the peaks
-                sound_background = _shift(sound_background)
-        print(f'\n====SOUND BACKGROUND SHAPE\n{sound_background.shape=}{np.abs(sound_background.max())=}\n{x.shape=}\n- - - -')
-        x = .9 * x + .1 * sound_background[:len_speech]
+        # generate 4
+        print('Generating AudioCraft')
+        back = [sound_generator.generate(
+                                        [scene]
+                                            )[0].detach().cpu().numpy()[0, :] for _ in range(4)]
+        
+        print([j.shape for j in back], len(back), 'BACK')
+        
+        # upsample to 24kHZ of StyleTTS
+        print('Resampling')
+        back = [audresample.resample(i,
+            original_rate=sound_generator.sample_rate,  # 16000
+            target_rate=24000
+            )[0, :] for i in back]
+        print('Cloning backgrounds')
+        # clone/elongate by 4x
+        back = [(_shift(np.concatenate([single_gen] * 4))) for single_gen in back]
+        
+        
+        # long ~30s
+        back = np.concatenate(back)
+        for _ in range(4):
+            back = _shift(back)
+        
+        # clone to exact len of TTS
+        n_repeat = len(x) // back.shape[0] + 2
+        
+        # Additional Repeat - Reach full length of TTS
+        print(f'Additional Repeat {n_repeat=}')
+        back = np.concatenate(n_repeat * [back])
+        back = _shift(back)
+        print(f'\n====SOUND BACKGROUND SHAPE\n{back.shape=}',
+              f'{np.abs(back.max())=}\n{x.shape=}')
+        x = .9 * x + .1 * back[:len(x)]
     else:
         print('sound_background = None')
     return x

landscape2soundscape.py

@@ -45,9 +45,9 @@ DESCRIPTIONS = [
     [
         '01_Schick_AII840_001.jpg',                               # image
         '01_Schick_AII840_001.txt',                               # text
-        'Statue in shire hill on autumn beach.',                  # audiocraft
+        'statue in shire ruisseau.',                  # audiocraft
         'G. Schick, Bildnis der Heinrike Dannecker, 1802', # cv2 puttext title
-        'en_US/m-ailabs_low#mary_ann',
+        'en_US/m-ailabs_low#judy_bieber', #'en_US/m-ailabs_low#mary_ann',
      ],  
     # 2
     [
@@ -87,7 +87,7 @@ DESCRIPTIONS = [
         '06_Menzel_AI900_001.txt',
         'Olive Seville',
         'A. Menzel, Bauplatz mit Weiden, 1846',
-        'en_US/m-ailabs_low#mary_ann',
+        'en_US/cmu-arctic_low#aup',
     ],
     # 7
     [
@@ -103,7 +103,7 @@ DESCRIPTIONS = [
         '08_Monet_AI1013_001.txt',
         'Mai flowers blossom picnic',
         'C. Monet, Sommertag, 1874',
-        'en_US/m-ailabs_low#mary_ann',
+        'en_US/cmu-arctic_low#axb',
     ],
     # 9
     [
@@ -119,7 +119,7 @@ DESCRIPTIONS = [
         '10_Boecklin_967648_NG2-80_001.txt',
         'Hades ades at it sisland',
         'A. Böcklin, Toteninsel, 1883',
-        'en_US/m-ailabs_low#mary_ann',
+        'en_US/cmu-arctic_low#jmk',
     ],
     # 11
     [
@@ -127,7 +127,7 @@ DESCRIPTIONS = [
         '11_Liebermann_NG4-94_001.txt',
         'Tavern at the waterfront',
         'M. Tiebermann, Gartenlokal An Der Havel. Nikolskoe, 1916',
-        'en_US/m-ailabs_low#mary_ann',
+        'en_US/cmu-arctic_low#ljm',
     ],
     # 12
     [

uc_spk_Landscape2Soundscape_Masterpieces_pics/01_Schick_AII840_001.txt

@@ -1 +1 @@
-After studying for four years with the legendary Parisian painter Jacques-Louis David, Christian Gottlieb Shic returned to Stuttgart in 1802. That year he began painting the portrait of Heinrike Dannecker the wife of his close friend and mentor Johannheinrich Dannecker. The home of the Danneckers known as the Danneckeray become a living hub for art lovers filled with ancient sculpture replicas. Schic portrait of Heinrike wasn’t any portrait as her relaxed pose, appalled to a clear confidenence of gaze reflects a new sense of freedom for the woman, one that arose in the wake of the French Revolution. It’s no accident that Heinrike’s outfit subtly nods to the colors of the French flag red, white, and blue. And the bouquet she holds, with bellflowers, roses, buttercups, and clover, stands out vividly against her deep blue dress, hinting at an effortless beauty. Lady Heinrike sits on a stone reminiscent of ancient pillars, her figure extending elegantly beyond the horizon. Behind her, the landscape ravels horizons and mountain ranges down to a tranquil river valley Interestingly, an unfinished version of this portrait—abandoned and trimmed by Schick himself—can still be found in the Staatsgalerie Stuttgart.
+After studying for four years with the legendary Parisian painter Jacques-Louis David, Christian Gottlieb Shick returned to Stuttgart in 1802. That year he began painting the portrait of Heinrike, the wife of his close friend and mentor Johannes Heinrich Dannecker. The home of the Danneckers was a living hub for art lovers filled with ancient sculpture replicas. Schic portrait of Heinrike wasn’t any portrait as her relaxed pose, appalled to a clear confidenence of gaze reflects a new sense of freedom for the woman, one that arose in the wake of the French Revolution. It’s no accident that Heinrike’s outfit subtly nods to the colors of the French flag red, white, and blue. And the bouquet she holds, with bellflowers, roses, buttercups, and clover, stands out vividly against her deep blue dress, hinting at an effortless beauty. Lady Heinrike sits on a stone reminiscent of ancient pillars, her figure extending elegantly beyond the horizon. Behind her, the landscape ravels horizons and mountain ranges down to a tranquil river valley Interestingly, an unfinished version of this ortrait—abandoned and trimmed by Shick himself—can still be found in the Staatsgalerie Stuttgart.