|
--- |
|
license: cc-by-nc-sa-4.0 |
|
--- |
|
|
|
**Restore missing RGB channels** |
|
|
|
Restore a missing channel of a RGB image by using ControlNet to guide image generation of Stable Diffusion to infer missing channel from the other two channels. |
|
* See accompanying discussion at [github.com - Channels RGB](https://github.com/lllyasviel/ControlNet/discussions/567) with detailed report and evaluations. |
|
* To restore images with missing channels you can use [this space](https://huggingface.co/spaces/GeroldMeisinger/channels). |
|
* For evaluation images see the corresponding .zip's at "files". |
|
* To run your own evaluations you can use [this script at gitlab.com](https://gitlab.com/-/snippets/3602096). |
|
|
|
# Training |
|
|
|
``` |
|
accelerate launch train_controlnet.py \ |
|
--pretrained_model_name_or_path="runwayml/stable-diffusion-v1-5" \ |
|
--train_batch_size=4 \ |
|
--gradient_accumulation_steps=8 \ |
|
--proportion_empty_prompts=0.5 |
|
--mixed_precision="fp16" \ |
|
--learning_rate=1e-5 \ |
|
--enable_xformers_memory_efficient_attention \ |
|
--use_8bit_adam \ |
|
--set_grads_to_none \ |
|
--seed=0 \ |
|
--num_train_epochs=2 |
|
``` |
|
|
|
# Image dataset |
|
|
|
* laion2B-en aesthetics>=6.5 dataset |
|
* --min_image_size 512 --max_aspect_ratio 2 --resize_mode="center_crop" --image_size 512 |
|
* Cleaned with `fastdup` default settings |
|
* Data augmented with right-left flipped images |
|
* Resulting in 214244 images |
|
* Set whole channel to 0 by alternating between R-G-B channels |
|
|
|
|
