README.md

---
pipeline_tag: text-to-image
license: other
license_name: faipl-1.0-sd
license_link: LICENSE
decoder:
- Disty0/sotediffusion-wuerstchen3-alpha1-decoder
---


# SoteDiffusion Wuerstchen3

Anime finetune of Würstchen V3.  
Currently is in early state in training.  
No commercial use thanks to StabilityAI.  

# Release Notes

 - Ran LLaVa on the images that has "english text" tag in it.  
   This adds `The text says "text"` tag.  
   If LLaVa has no idea what the text is, it describes the image instead.  

<style>
.image {
    float: left;
    margin-left: 10px;
}
</style>

<table>
<img class="image" src="https://cdn-uploads.huggingface.co/production/uploads/6456af6195082f722d178522/pFev-xGFut0o3qlZQJpwb.png" width="320">
<img class="image" src="https://cdn-uploads.huggingface.co/production/uploads/6456af6195082f722d178522/adhnnnmRBkTULFNl9AfT2.png" width="320">
</table>

# UI Guide

## SD.Next
URL: https://github.com/vladmandic/automatic/

Go to Models -> Huggingface and type `Disty0/sotediffusion-wuerstchen3-alpha2-decoder` into the model name and press download.  
Load `Disty0/sotediffusion-wuerstchen3-alpha2-decoder` after the download process is complete.  

Prompt:  
```
very aesthetic, best quality, newest,
```

Negative Prompt:  
```
very displeasing, worst quality, oldest, monochrome, sketch, realistic,
```

Parameters:  
Sampler: Default  

Steps: 30 or 40  
Refiner Steps: 10  

CFG: 8  
Secondary CFG: 1 or 1.2  

Resolution: 1024x1536, 2048x1152  
Anything works as long as it's a multiply of 128.


## ComfyUI

Please refer to CivitAI: https://civitai.com/models/353284  


# Code Example

```shell
pip install diffusers
```

```python
import torch
from diffusers import StableCascadeCombinedPipeline

device = "cuda"
dtype = torch.bfloat16
model = "Disty0/sotediffusion-wuerstchen3-alpha2-decoder"

pipe = StableCascadeCombinedPipeline.from_pretrained(model, torch_dtype=dtype)

# send everything to the gpu:
pipe = pipe.to(device, dtype=dtype)
pipe.prior_pipe = pipe.prior_pipe.to(device, dtype=dtype)

# or enable model offload to save vram:
# pipe.enable_model_cpu_offload()



prompt = "1girl, solo, cowboy shot, straight hair, looking at viewer, hoodie, indoors, slight smile, casual, furniture, doorway, very aesthetic, best quality, newest,"
negative_prompt = "very displeasing, worst quality, oldest, monochrome, sketch, realistic,"

output = pipe(
    width=1024,
    height=1536,
    prompt=prompt,
    negative_prompt=negative_prompt,
    decoder_guidance_scale=1.0,
    prior_guidance_scale=8.0,
    prior_num_inference_steps=40,
    output_type="pil",
    num_inference_steps=10
).images[0]

## do something with the output image
```


## Training Status:

**GPU used for training**: 1x AMD RX 7900 XTX 24GB  
**GPU Hours**: 250 (Accumulative starting from alpha1)  

| dataset name | training done | remaining |
|---|---|---|
| **newest** | 010 | 221 |
| **recent** | 010 | 162 |
| **mid** | 010 | 114 |
| **early** | 010 | 060 |
| **oldest** | 010 | 010 |
| **pixiv** | 010 | 032 |
| **visual novel cg** | 010 | 018 |
| **anime wallpaper** | 010 | 003 |
| **Total** | 88 | 620 |

**Note**: chunks starts from 0 and there are 8000 images per chunk  


## Dataset:

**GPU used for captioning**: 1x Intel ARC A770 16GB  
**GPU Hours**: 350  

**Model used for captioning**: SmilingWolf/wd-swinv2-tagger-v3  
**Command:**  
```
python /mnt/DataSSD/AI/Apps/kohya_ss/sd-scripts/finetune/tag_images_by_wd14_tagger.py --model_dir "/mnt/DataSSD/AI/models/wd14_tagger_model" --repo_id "SmilingWolf/wd-swinv2-tagger-v3" --recursive --remove_underscore --use_rating_tags --character_tags_first --character_tag_expand --append_tags --onnx --caption_separator ", " --general_threshold 0.35 --character_threshold 0.50 --batch_size 4 --caption_extension ".txt" ./
```


| dataset name | total images | total chunk |
|---|---|---|
| **newest** | 1.848.331 | 232 |
| **recent** | 1.380.630 | 173 |
| **mid** | 993.227 | 125 |
| **early** | 566.152 | 071 |
| **oldest** | 160.397 | 021 |
| **pixiv** | 343.614 | 043 |
| **visual novel cg** | 231.358 | 029 |
| **anime wallpaper** | 104.790 | 014 |
| **Total** | 5.628.499 | 708 |

**Note**:  
 - Smallest size is 1280x600 | 768.000 pixels
 - Deduped based on image similarity using czkawka-cli


## Tags:

Model is trained with random tag order but this is the order in the dataset if you are interested:  
```
aesthetic tags, quality tags, date tags, custom tags, rating tags, character, series, rest of the tags
```

### Date:

| tag | date |
|---|---|
| **newest** | 2022 to 2024 |
| **recent** | 2019 to 2021 |
| **mid** | 2015 to 2018 |
| **early** | 2011 to 2014 |
| **oldest** | 2005 to 2010 |

### Aesthetic Tags:
**Model used**: shadowlilac/aesthetic-shadow-v2

| score greater than | tag | count |
|---|---|---|
| **0.90** | extremely aesthetic | 125.451 |
| **0.80** | very aesthetic | 887.382 |
| **0.70** | aesthetic | 1.049.857 |
| **0.50** | slightly aesthetic | 1.643.091 |
| **0.40** | not displeasing | 569.543 |
| **0.30** | not aesthetic | 445.188 |
| **0.20** | slightly displeasing | 341.424 |
| **0.10** | displeasing | 237.660 |
| **rest of them** | very displeasing | 328.712 |

### Quality Tags:
**Model used**: https://huggingface.co/hakurei/waifu-diffusion-v1-4/blob/main/models/aes-B32-v0.pth

| score greater than | tag | count |
|---|---|---|
| **0.980** | best quality | 1.270.447 |
| **0.900** | high quality | 498.244 |
| **0.750** | great quality | 351.006 |
| **0.500** | medium quality | 366.448 |
| **0.250** | normal quality | 368.380 |
| **0.125** | bad quality | 279.050 |
| **0.025** | low quality | 538.958 |
| **rest of them** | worst quality | 1.955.966 |

## Rating Tags

| tag | count |
|---|---|
| **general** | 1.416.451 |
| **sensitive** | 3.447.664 |
| **nsfw** | 427.459 |
| **explicit nsfw** | 336.925 |

## Custom Tags:

| dataset name | custom tag |
|---|---|
| **image boards** | date, |
| **pixiv** | art by Display_Name, |
| **visual novel cg** | Full_VN_Name (short_3_letter_name), visual novel cg, |
| **anime wallpaper** | date, anime wallpaper, |

## Training Parameters:
**Software used**: Kohya SD-Scripts with Stable Cascade branch  
https://github.com/kohya-ss/sd-scripts/tree/stable-cascade  

**Base model**: Disty0/sote-diffusion-cascade-alpha0  
### Command:
```shell
#!/bin/sh

CURRENT=$1
CURRENT_SUB=$2

PAST=$3
PAST_SUB=$4

LD_PRELOAD=/usr/lib/libtcmalloc.so.4 accelerate launch --mixed_precision fp16 --num_cpu_threads_per_process 1 stable_cascade_train_stage_c.py \
--mixed_precision fp16 \
--save_precision fp16 \
--full_fp16 \
--sdpa \
--gradient_checkpointing \
--train_text_encoder \
--resolution "1024,1024" \
--train_batch_size 2 \
--gradient_accumulation_steps 8 \
--learning_rate 1e-5 \
--learning_rate_te1 1e-5 \
--lr_scheduler constant_with_warmup \
--lr_warmup_steps 100 \
--optimizer_type adafactor \
--optimizer_args "scale_parameter=False" "relative_step=False" "warmup_init=False" \
--max_grad_norm 0 \
--token_warmup_min 1 \
--token_warmup_step 0 \
--shuffle_caption \
--caption_separator ", " \
--caption_dropout_rate 0 \
--caption_tag_dropout_rate 0 \
--caption_dropout_every_n_epochs 0 \
--dataset_repeats 1 \
--save_state \
--save_every_n_steps 256 \
--sample_every_n_steps 64 \
--max_token_length 225 \
--max_train_epochs 1 \
--caption_extension ".txt" \
--max_data_loader_n_workers 2 \
--persistent_data_loader_workers \
--enable_bucket \
--min_bucket_reso 256 \
--max_bucket_reso 4096 \
--bucket_reso_steps 64 \
--bucket_no_upscale \
--log_with tensorboard \
--output_name sotediffusion-wr3_3b \
--train_data_dir /mnt/DataSSD/AI/anime_image_dataset/combined/combined-$CURRENT/$CURRENT_SUB \
--in_json /mnt/DataSSD/AI/anime_image_dataset/combined/combined-$CURRENT/$CURRENT_SUB.json \
--output_dir /mnt/DataSSD/AI/SoteDiffusion/Wuerstchen3/sotediffusion-wr3_3b-$CURRENT/$CURRENT_SUB \
--logging_dir /mnt/DataSSD/AI/SoteDiffusion/Wuerstchen3/sotediffusion-wr3_3b-$CURRENT/$CURRENT_SUB/logs \
--resume /mnt/DataSSD/AI/SoteDiffusion/Wuerstchen3/sotediffusion-wr3_3b-$PAST/$PAST_SUB/sotediffusion-wr3_3b-state \
--stage_c_checkpoint_path /mnt/DataSSD/AI/SoteDiffusion/Wuerstchen3/sotediffusion-wr3_3b-$PAST/$PAST_SUB/sotediffusion-wr3_3b.safetensors \
--text_model_checkpoint_path /mnt/DataSSD/AI/SoteDiffusion/Wuerstchen3/sotediffusion-wr3_3b-$PAST/$PAST_SUB/sotediffusion-wr3_3b_text_model.safetensors \
--effnet_checkpoint_path /mnt/DataSSD/AI/models/wuerstchen3/effnet_encoder.safetensors \
--previewer_checkpoint_path /mnt/DataSSD/AI/models/wuerstchen3/previewer.safetensors \
--sample_prompts /mnt/DataSSD/AI/SoteDiffusion/Wuerstchen3/config/sotediffusion-prompt.txt
```


## Limitations and Bias

### Bias

- This model is intended for anime illustrations.  
  Realistic capabilites are not tested at all.  

### Limitations

- Can fall back to realistic.  
  Add "realistic" tag to the negatives when this happens.  
- Far shot eyes can be bad.  
- Anatomy and hands can be bad.  
- Still in active training.  


## License

SoteDiffusion models falls under [Fair AI Public License 1.0-SD](https://freedevproject.org/faipl-1.0-sd/) license, which is compatible with Stable Diffusion models’ license. Key points:

1. **Modification Sharing:** If you modify SoteDiffusion models, you must share both your changes and the original license.
2. **Source Code Accessibility:** If your modified version is network-accessible, provide a way (like a download link) for others to get the source code. This applies to derived models too.
3. **Distribution Terms:** Any distribution must be under this license or another with similar rules.
4. **Compliance:** Non-compliance must be fixed within 30 days to avoid license termination, emphasizing transparency and adherence to open-source values.

**Notes**: Anything not covered by Fair AI license is inherited from Stability AI Non-Commercial license which is named as LICENSE_INHERIT. Meaning, still no commercial use of any kind.