File size: 2,209 Bytes
7a4ce75 f18f233 7a4ce75 9fd9218 7a4ce75 9fd9218 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 |
---
library_name: transformers
license: apache-2.0
datasets:
- merve/vqav2-small
---
# Idefics3Llama Fine-tuned using QLoRA on VQAv2
- This is the [Idefics3Llama](https://huggingface.co/HuggingFaceM4/Idefics3-8B-Llama3) model QLoRA fine-tuned on a very small part of [VQAv2](https://huggingface.co/datasets/merve/vqav2-small) dataset.
- Find the fine-tuning notebook [here](https://github.com/merveenoyan/smol-vision/blob/main/Idefics_FT.ipynb).
## Usage
You can load and use this model as follows.
```python
from transformers import Idefics3ForConditionalGeneration, AutoProcessor
peft_model_id = "idefics3-llama-vqav2/checkpoint-535"
base_model_id = "HuggingFaceM4/Idefics3-8B-Llama3"
processor = AutoProcessor.from_pretrained(base_model_id)
model = Idefics3ForConditionalGeneration.from_pretrained(base_model_id)
model.load_adapter(peft_model_id).to("cuda")
```
This model was conditioned on a prompt "Answer briefly.".
```python
from PIL import Image
import requests
from transformers.image_utils import load_image
DEVICE = "cuda"
image = load_image("https://huggingface.co/spaces/merve/OWLSAM2/resolve/main/buddha.JPG")
messages = [
{
"role": "user",
"content": [
{"type": "text", "text": "Answer briefly."},
{"type": "image"},
{"type": "text", "text": "Which country is this located in?"}
]
}
]
text = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=text, images=image, return_tensors="pt", padding=True).to("cuda")
```
We can infer.
```python
generated_ids = model.generate(**inputs, max_new_tokens=500)
generated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True)
print(generated_texts)
##['User: Answer briefly.<row_1_col_1><row_1_col_2><row_1_col_3><row_1_col_4>\n<row_2_col_1>
# <row_2_col_2><row_2_col_3><row_2_col_4>\n<row_3_col_1><row_3_col_2><row_3_col_3>
# <row_3_col_4>\n\n<global-img>Which country is this located in?\nAssistant: thailand\nAssistant: thailand']
``` |