|
--- |
|
license: openrail |
|
--- |
|
|
|
|
|
# gpt2-azerbaijani-smallv0 model for text generation |
|
|
|
|
|
## Introduction |
|
|
|
gpt2-azerbaijani-smallv0 is a state-of-the-art language model for Azerbaijani based on the GPT-2 small model. |
|
|
|
It was trained on Azerbaijani Wikipedia using Transfer Learning and Fine-tuning techniques in ~ 29 hours, on one GPU - 1 x NVIDIA Tesla K80. |
|
|
|
## Model |
|
|
|
| Model | #params | Model file (pt) | Arch. | Training /Validation data (text) | |
|
|-------------------------|---------|--------------------|-------------|------------------------------------------| |
|
| gpt2-azerbaijani-smallv0| 124M | 652 | GPT-2 small | Azerbaijani Wikipedia (110k articles / 19k articles) | |
|
|
|
|
|
epoches - 3, loss - 5.17, accuracy - 23.99%, perplexity - 95.88 |
|
|
|
|
|
## How to use GPorTuguese-2 with HuggingFace (PyTorch) |
|
|
|
The following code use PyTorch. |
|
|
|
```python |
|
import torch |
|
from transformers import GPT2LMHeadModel, AutoTokenizer, AutoModelWithLMHead |
|
|
|
tokenizer = AutoTokenizer.from_pretrained("nijatzeynalov/gpt2-azerbaijani-small") |
|
tokenizer.model_max_length=1024 |
|
|
|
model_state_dict = torch.load('GPT2_pt_3epoch_lr2e-3.pth', map_location=torch.device('cpu')) |
|
model = GPT2LMHeadModel.from_pretrained('gpt2', state_dict=model_state_dict) |
|
|
|
model.eval() |
|
|
|
text = "Your prompt here" |
|
inputs = tokenizer(text, return_tensors="pt") |
|
|
|
sample_outputs = model.generate(inputs.input_ids, |
|
pad_token_id=50256, |
|
do_sample=True, |
|
max_length=20, |
|
top_k=10, |
|
num_return_sequences=1) |
|
|
|
# generated sequence |
|
for i, sample_output in enumerate(sample_outputs): |
|
print(">> Generated text {}\n\n{}".format(i+1, tokenizer.decode(sample_output.tolist()))) |
|
``` |
|
|
|
## Bias |
|
|
|
The training data used for this model come from Azerbaijani Wikipedia. We know it contains a lot of unfiltered content from the internet, which is far from neutral. As the openAI team themselves point out in their model card: |
|
|
|
> Because large-scale language models like GPT-2 do not distinguish fact from fiction, we don’t support use-cases that require the generated text to be true. Additionally, language models like GPT-2 reflect the biases inherent to the systems they were trained on, so we do not recommend that they be deployed into systems that interact with humans > unless the deployers first carry out a study of biases relevant to the intended use-case. We found no statistically significant difference in gender, race, and religious bias probes between 774M and 1.5B, implying all versions of GPT-2 should be approached with similar levels of caution around use cases that are sensitive to biases around human attributes. |
|
|
|
## Limitations |
|
|
|
__This model was developed for the purpose of research for the application of the GPT-2 model to the Azerbaijani language, and the results it produces are of very low quality due to resource limitations, the current version is not recommended for use in commercial projects.__ |
|
|
|
Since my current resources are limited, I will return to this model again, I plan to improve the results: |
|
|
|
* Add more train data in Azerbaijani language; I plan to find and add 500k+ articles using various resources, not just wikipedia. |
|
* Clean the Train dataset better; Currently, due to lack of resources, cleaning is hardly done. |
|
* Running different experiments using a more powerful GPU. Only 1cycle policy for fine tuning technique was tested. |
|
* Increase the number of Epoch; With the current GPU (GPU - 1 x NVIDIA Tesla K80), 1 epoch lasts about ~9 hours ($0.90/hr). Considering the goal of the project and other resources, I found it acceptable to stop at 3 epochs. |
|
|
|
## Author |
|
|
|
Azerbaijani GPT-2 small was trained and evaluated by [Nijat Zeynalov](https://www.linkedin.com/in/nijat-zeynalov-064163142/). |
|
|
|
|
