File size: 6,014 Bytes
4d03da9 8eb02a3 4d03da9 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 |
---
license: mit
inference: false
base_model: naver-clova-ix/donut-base
tags:
- donut
- image-to-text
- vision
model-index:
- name: donut-dr-matriculas-ocr
results:
- task:
type: image-to-text
name: Image to text
metrics:
- type: loss
value: 0.0563
name: Final loss (50 epochs)
- type: accuracy
value: 0.724689
name: F1 Accuracy (Val)
- type: accuracy
value: 0.923603
name: F1 Accuracy (Train)
- type: edit distance
value: 0.914544
name: ED (Val)
- type: edit distance
value: 0.971895
name: ED (Train)
metrics:
- accuracy
datasets:
- propietary/matriculas
pipeline_tag: image-to-text
---
# Donut 🍩 for DR Matriculas (Donut-DR-matriculas-OCR)
Donut model was introduced in the paper [OCR-free Document Understanding Transformer](https://arxiv.org/abs/2111.15664) by Geewok et al. and first released in [this repository](https://github.com/clovaai/donut).
## === Matriculas OCR V1 ===
This model is a finetune of the [donut base model](https://huggingface.co/naver-clova-ix/donut-base/) on a propietary dataset. Its purpose is to efficiently extract text from the dominican official vehicle registration documents.
This propietary dataset was manually corrected, and we prepared the teacher forcing (ground truth) data with the images and json lines. The license for the V1 model is **mit**, available under the MIT license.
It achieves the following results on the evaluation set:
* Loss: 0.0563
* Edit distance: 0.914544
* F1 accuracy: 0.724689
The task_prompt has been changed to ``<s_matricula>`` for the V1.
The focus for the next or future version, will be to collect a better an larger dataset for training.
## Model description
Donut consists of a vision encoder (Swin Transformer) and a text decoder (BART). Given an image, the encoder first encodes the image into a tensor of embeddings (of shape batch_size, seq_len, hidden_size), after which the decoder autoregressively generates text, conditioned on the encoding of the encoder.
### How to use
```python
import torch
import re
from PIL import Image
from transformers import DonutProcessor
#from transformers import VisionEncoderDecoderModel
import warnings
warnings.filterwarnings("ignore")
from sconf import Config
from donut import DonutConfig, DonutModel
config = Config(default="./config.yaml")
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
processor = DonutProcessor.from_pretrained("marzanconsulting/donut-dr-matriculas-ocr")
model = DonutModel.from_pretrained(
"marzanconsulting/donut-dr-matriculas-ocr",
input_size=config.input_size,
max_length=config.max_length,
align_long_axis=config.align_long_axis,
ignore_mismatched_sizes=True,
)
model.to(device)
def load_and_preprocess_image(image_path: str, processor):
"""
Load an image and preprocess it for the model.
"""
image = Image.open(image_path).convert("RGB")
pixel_values = processor(image, return_tensors="pt").pixel_values
return pixel_values
def generate_text_from_image(model, image_path: str, processor, device):
"""
Generate text from an image using the trained model.
"""
# Load and preprocess the image
pixel_values = load_and_preprocess_image(image_path, processor)
pixel_values = pixel_values.to(device)
decoder_input_ids = processor.tokenizer(task_prompt="<s_matricula>",
add_special_tokens=False,
return_tensors="pt").input_ids
decoded_text = model.inference(image_tensors=pixel_values,
prompt_tensors=decoder_input_ids)["predictions"][0]
return decoded_text
# Example usage
image_path = "path_to_your_image" # Replace with your image path
extracted_text = generate_text_from_image(model, image_path, processor, device)
print("Extracted Text:", extracted_text)
```
Refer to the [documentation](https://huggingface.co/docs/transformers/main/en/model_doc/donut) for more code examples.
## Intended uses & limitations
This fine-tuned model is specifically designed for extracting text from dominican vehicle registration (matriculas) documents, and may not perform optimally on other types of documents. The dataset used is still suboptimal (numerous errors are still there), thus, this model will need to be retrained later to improve its performance.
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 5
- eval_batch_size: 1
- seed: 2022
- optimizer: AdamW with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 300
- num_epochs: 50
- weight_decay: 0.01
### Framework versions
- Transformers 4.25.1
- Timm 0.6.13
- Pytorch-lightning 1.6.4
- Donut 1.0.9
If you want to support me, you can [here](https://www.marzanconsulting.com/).
### BibTeX entry and citation info for DONUT
```bibtex
@article{DBLP:journals/corr/abs-2111-15664,
author = {Geewook Kim and
Teakgyu Hong and
Moonbin Yim and
Jinyoung Park and
Jinyeong Yim and
Wonseok Hwang and
Sangdoo Yun and
Dongyoon Han and
Seunghyun Park},
title = {Donut: Document Understanding Transformer without {OCR}},
journal = {CoRR},
volume = {abs/2111.15664},
year = {2021},
url = {https://arxiv.org/abs/2111.15664},
eprinttype = {arXiv},
eprint = {2111.15664},
timestamp = {Thu, 02 Dec 2021 10:50:44 +0100},
biburl = {https://dblp.org/rec/journals/corr/abs-2111-15664.bib},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
```
|