metadata

license: other
base_model: black-forest-labs/FLUX.1-dev
tags:
  - flux
  - flux-diffusers
  - text-to-image
  - diffusers
  - simpletuner
  - not-for-all-audiences
  - lora
  - template:sd-lora
  - lycoris
inference: true
widget:
  - text: unconditional (blank prompt)
    parameters:
      negative_prompt: blurry, cropped, ugly
    output:
      url: ./assets/image_0_0.png
  - text: >-
      In the image, PhongKamy is portrayed in a thoughtful pose. He is seated on
      a simple, modern white platform against a soft, light gray background that
      enhances the serene atmosphere. PhongKamy has neat, slightly wavy hair
      styled closely on the sides and with more volume on top. He is wearing a
      crisp, white dress shirt, which is neatly buttoned up and has long sleeves
      that are slightly rolled back at the wrists, giving a relaxed yet polished
      look. His expression is contemplative, with a slight smile playing at the
      corners of his mouth. PhongKamy holds a pair of glasses in his right hand,
      delicately resting them near his chin, adding an air of sophistication.
      His left arm is positioned casually on his knee, showing a relaxed
      demeanor. He wears tailored black trousers that contrast well with his
      white shirt. Overall, the composition focuses on PhongKamy's thoughtful
      expression and stylish appearance, creating a captivating and elegant
      portrait.
    parameters:
      negative_prompt: blurry, cropped, ugly
    output:
      url: ./assets/image_1_0.png
  - text: >-
      In the image, PhongKamy stands confidently against a light gray
      background. He is wearing a crisp white shirt that is neatly buttoned up,
      exuding a professional appearance. His glasses add a touch of
      sophistication, framing his face. PhongKamy holds a vibrant red passport
      in one hand, which features a prominent compass design on the cover,
      suggesting a theme of travel or adventure. His other hand is casually
      placed in his pocket, and he has a relaxed yet poised expression, gazing
      slightly upward, as if contemplating future journeys. Below his shirt, he
      wears dark trousers, complemented by a brown belt with a shiny buckle,
      adding a stylish contrast to his outfit. The overall composition captures
      a sense of ambition and readiness for exploration.
    parameters:
      negative_prompt: blurry, cropped, ugly
    output:
      url: ./assets/image_2_0.png
  - text: >-
      In the image, PhongKamy is posed in a relaxed manner, conveying a sense of
      calm and confidence. He is wearing a crisp white shirt that contrasts
      nicely against his darker pants. The shirt has a classic collar and is
      unbuttoned at the top, giving it a casual yet polished look. PhongKamy's
      dark hair is neatly styled, with a slight wave, enhancing his youthful
      appearance. He is seated with one leg crossed over the other, and he holds
      a pair of clear, stylish glasses in one hand, suggesting a thoughtful
      demeanor. His smile is warm and inviting, reflecting a friendly
      personality. The background is a soft gray, which allows PhongKamy to
      stand out as the focal point of the image. The lighting is soft and
      diffused, creating a flattering effect on his features and adding to the
      overall serene atmosphere of the portrait.
    parameters:
      negative_prompt: blurry, cropped, ugly
    output:
      url: ./assets/image_3_0.png

simpletuner-lora

This is a LyCORIS adapter derived from black-forest-labs/FLUX.1-dev.

The main validation prompt used during training was:

In the image, PhongKamy is posed in a relaxed manner, conveying a sense of calm and confidence. He is wearing a crisp white shirt that contrasts nicely against his darker pants. The shirt has a classic collar and is unbuttoned at the top, giving it a casual yet polished look. PhongKamy's dark hair is neatly styled, with a slight wave, enhancing his youthful appearance. He is seated with one leg crossed over the other, and he holds a pair of clear, stylish glasses in one hand, suggesting a thoughtful demeanor. His smile is warm and inviting, reflecting a friendly personality. The background is a soft gray, which allows PhongKamy to stand out as the focal point of the image. The lighting is soft and diffused, creating a flattering effect on his features and adding to the overall serene atmosphere of the portrait.

Validation settings

CFG: 3.0
CFG Rescale: 0.0
Steps: 20
Sampler: FlowMatchEulerDiscreteScheduler
Seed: 42
Resolution: 1024x1024
Skip-layer guidance:

Note: The validation settings are not necessarily the same as the training settings.

You can find some example images in the following gallery:

Prompt
unconditional (blank prompt)

Negative Prompt
blurry, cropped, ugly

Prompt
In the image, PhongKamy is portrayed in a thoughtful pose. He is seated on a simple, modern white platform against a soft, light gray background that enhances the serene atmosphere. PhongKamy has neat, slightly wavy hair styled closely on the sides and with more volume on top. He is wearing a crisp, white dress shirt, which is neatly buttoned up and has long sleeves that are slightly rolled back at the wrists, giving a relaxed yet polished look. His expression is contemplative, with a slight smile playing at the corners of his mouth. PhongKamy holds a pair of glasses in his right hand, delicately resting them near his chin, adding an air of sophistication. His left arm is positioned casually on his knee, showing a relaxed demeanor. He wears tailored black trousers that contrast well with his white shirt. Overall, the composition focuses on PhongKamy's thoughtful expression and stylish appearance, creating a captivating and elegant portrait.

Negative Prompt
blurry, cropped, ugly

Prompt
In the image, PhongKamy stands confidently against a light gray background. He is wearing a crisp white shirt that is neatly buttoned up, exuding a professional appearance. His glasses add a touch of sophistication, framing his face. PhongKamy holds a vibrant red passport in one hand, which features a prominent compass design on the cover, suggesting a theme of travel or adventure. His other hand is casually placed in his pocket, and he has a relaxed yet poised expression, gazing slightly upward, as if contemplating future journeys. Below his shirt, he wears dark trousers, complemented by a brown belt with a shiny buckle, adding a stylish contrast to his outfit. The overall composition captures a sense of ambition and readiness for exploration.

Negative Prompt
blurry, cropped, ugly

Prompt
In the image, PhongKamy is posed in a relaxed manner, conveying a sense of calm and confidence. He is wearing a crisp white shirt that contrasts nicely against his darker pants. The shirt has a classic collar and is unbuttoned at the top, giving it a casual yet polished look. PhongKamy's dark hair is neatly styled, with a slight wave, enhancing his youthful appearance. He is seated with one leg crossed over the other, and he holds a pair of clear, stylish glasses in one hand, suggesting a thoughtful demeanor. His smile is warm and inviting, reflecting a friendly personality. The background is a soft gray, which allows PhongKamy to stand out as the focal point of the image. The lighting is soft and diffused, creating a flattering effect on his features and adding to the overall serene atmosphere of the portrait.

Negative Prompt
blurry, cropped, ugly

The text encoder was not trained. You may reuse the base model text encoder for inference.

Training settings

Training epochs: 0
Training steps: 10000
Learning rate: 0.0001
- Learning rate schedule: polynomial
- Warmup steps: 100
Max grad norm: 1.0
Effective batch size: 1
- Micro-batch size: 1
- Gradient accumulation steps: 1
- Number of GPUs: 1
Gradient checkpointing: True
Prediction type: flow-matching (extra parameters=['shift=3', 'flux_guidance_mode=constant', 'flux_guidance_value=1.0', 'flow_matching_loss=compatible'])
Optimizer: adamw_bf16
Trainable parameter precision: Pure BF16
Caption dropout probability: 10.0%

LyCORIS Config:

{
    "algo": "lokr",
    "multiplier": 1.0,
    "linear_dim": 10000,
    "linear_alpha": 1,
    "factor": 16,
    "apply_preset": {
        "target_module": [
            "Attention",
            "FeedForward"
        ],
        "module_algo_map": {
            "Attention": {
                "factor": 16
            },
            "FeedForward": {
                "factor": 8
            }
        }
    }
}

Datasets

behitek-subject

Repeats: 1000
Total number of images: 22
Total number of aspect buckets: 1
Resolution: 1.048576 megapixels
Cropped: True
Crop style: center
Crop aspect: square
Used for regularisation data: No

behitek-subject-512

Repeats: 1000
Total number of images: 22
Total number of aspect buckets: 1
Resolution: 0.262144 megapixels
Cropped: True
Crop style: center
Crop aspect: square
Used for regularisation data: No

Inference

import torch
from diffusers import DiffusionPipeline
from lycoris import create_lycoris_from_weights


def download_adapter(repo_id: str):
    import os
    from huggingface_hub import hf_hub_download
    adapter_filename = "pytorch_lora_weights.safetensors"
    cache_dir = os.environ.get('HF_PATH', os.path.expanduser('~/.cache/huggingface/hub/models'))
    cleaned_adapter_path = repo_id.replace("/", "_").replace("\\", "_").replace(":", "_")
    path_to_adapter = os.path.join(cache_dir, cleaned_adapter_path)
    path_to_adapter_file = os.path.join(path_to_adapter, adapter_filename)
    os.makedirs(path_to_adapter, exist_ok=True)
    hf_hub_download(
        repo_id=repo_id, filename=adapter_filename, local_dir=path_to_adapter
    )

    return path_to_adapter_file
    
model_id = 'black-forest-labs/FLUX.1-dev'
adapter_repo_id = 'phongmt184172/simpletuner-lora'
adapter_filename = 'pytorch_lora_weights.safetensors'
adapter_file_path = download_adapter(repo_id=adapter_repo_id)
pipeline = DiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.bfloat16) # loading directly in bf16
lora_scale = 1.0
wrapper, _ = create_lycoris_from_weights(lora_scale, adapter_file_path, pipeline.transformer)
wrapper.merge_to()

prompt = "In the image, PhongKamy is posed in a relaxed manner, conveying a sense of calm and confidence. He is wearing a crisp white shirt that contrasts nicely against his darker pants. The shirt has a classic collar and is unbuttoned at the top, giving it a casual yet polished look. PhongKamy's dark hair is neatly styled, with a slight wave, enhancing his youthful appearance. He is seated with one leg crossed over the other, and he holds a pair of clear, stylish glasses in one hand, suggesting a thoughtful demeanor. His smile is warm and inviting, reflecting a friendly personality. The background is a soft gray, which allows PhongKamy to stand out as the focal point of the image. The lighting is soft and diffused, creating a flattering effect on his features and adding to the overall serene atmosphere of the portrait."


## Optional: quantise the model to save on vram.
## Note: The model was quantised during training, and so it is recommended to do the same during inference time.
from optimum.quanto import quantize, freeze, qint8
quantize(pipeline.transformer, weights=qint8)
freeze(pipeline.transformer)
    
pipeline.to('cuda' if torch.cuda.is_available() else 'mps' if torch.backends.mps.is_available() else 'cpu') # the pipeline is already in its target precision level
image = pipeline(
    prompt=prompt,
    num_inference_steps=20,
    generator=torch.Generator(device='cuda' if torch.cuda.is_available() else 'mps' if torch.backends.mps.is_available() else 'cpu').manual_seed(42),
    width=1024,
    height=1024,
    guidance_scale=3.0,
).images[0]
image.save("output.png", format="PNG")