Ananya8154
commited on
Commit
•
1bad355
1
Parent(s):
ae0ebd8
Update README.md
Browse files
README.md
CHANGED
@@ -44,45 +44,7 @@ The dataset consists of 3k instances of image-text pairs. Given below is sample
|
|
44 |
|
45 |
## How to use?
|
46 |
|
47 |
-
|
48 |
-
Using Transformers:
|
49 |
-
```python
|
50 |
-
from PIL import Image
|
51 |
-
import torch
|
52 |
-
from transformers import AutoProcessor, LlavaForConditionalGeneration
|
53 |
-
|
54 |
-
model_id = "Ananya8154/Llava-v1.5-7b-FineTuned-Historic-Art-dataset/llava-ftmodel"
|
55 |
-
model = LlavaForConditionalGeneration.from_pretrained(
|
56 |
-
model_id,
|
57 |
-
torch_dtype=torch.float16,
|
58 |
-
low_cpu_mem_usage=True,
|
59 |
-
).to(0)
|
60 |
-
|
61 |
-
processor = AutoProcessor.from_pretrained(model_id)
|
62 |
-
|
63 |
-
conversation = [
|
64 |
-
{
|
65 |
-
{
|
66 |
-
"from": "human",
|
67 |
-
"value": "What is this image?"
|
68 |
-
},
|
69 |
-
{
|
70 |
-
"from": "gpt",
|
71 |
-
"value": ""
|
72 |
-
}
|
73 |
-
}
|
74 |
-
]
|
75 |
-
prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
|
76 |
-
|
77 |
-
image_file = "test.jpg" # You can add image file path here # Use only Historical art image # or use the sample image 'test.jpg'
|
78 |
-
raw_image = Image.open(image_file)
|
79 |
-
inputs = processor(prompt, raw_image, return_tensors='pt').to(0, torch.float16)
|
80 |
-
|
81 |
-
output = model.generate(**inputs, max_new_tokens=200, do_sample=False)
|
82 |
-
print(processor.decode(output[0][2:], skip_special_tokens=True))
|
83 |
-
```
|
84 |
-
|
85 |
-
### OR
|
86 |
|
87 |
The folder 'llava-v1.5-7b-task-lora' contains the lora weights and the folder 'llava-ftmodel' contains the merged model weights and configurations.
|
88 |
- To use the model:
|
|
|
44 |
|
45 |
## How to use?
|
46 |
|
47 |
+
**Note** - Don't use the model with the transformers 'Use this model' on huggingface, alternatively follow the belows step wise approach for inferencing this model.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
48 |
|
49 |
The folder 'llava-v1.5-7b-task-lora' contains the lora weights and the folder 'llava-ftmodel' contains the merged model weights and configurations.
|
50 |
- To use the model:
|