|
--- |
|
language: |
|
- vi |
|
thumbnail: "url to a thumbnail used in social sharing" |
|
tags: |
|
- News |
|
- Language model |
|
- GPT2 |
|
datasets: |
|
- Private Vietnamese News dataset |
|
metrics: |
|
- rouge |
|
- wer |
|
|
|
--- |
|
|
|
|
|
# GPT-2 Fine-tuning With Vietnamese News |
|
## Model description |
|
A Fine-tuned Vietnamese GPT2 model which can generate Vietnamese news based on context (category + headline), based on the Vietnamese Wiki GPT2 pretrained model (https://huggingface.co/danghuy1999/gpt2-viwiki) |
|
|
|
## Github |
|
- https://github.com/Tuan-Lee-23/Vietnamese-News-Generative-Model |
|
|
|
## Purpose |
|
This model was made only for fun and experimental study. However, It gives impressive results |
|
Most of the generative news are fake with unconfirmed information. Honestly, I feel fun about this project =)) |
|
|
|
## Dataset |
|
The dataset is about 30k Vietnamese news dataset from website thanhnien.vn |
|
|
|
## Result |
|
- Train Loss: 2.3 |
|
- Val loss: 2.5 |
|
- Rouge F1: 0.556 |
|
- Word error rate: 1.08 |
|
|
|
## Deployment |
|
- You can run the model deployment in this Colab's [link](https://colab.research.google.com/drive/1ITnYPnngd_aqkFB2A5IhzSsX4jQSPOR1?usp=sharing) |
|
- Then go to this link: https://gptvn.loca.lt |
|
- You can choose any categories and give it some text for the headline, then generate. There we go |
|
- P/s: I've already tried to deploy my model on Streamlit's cloud, but It was always being broken due to out of memory |
|
|
|
|
|
## Example usage |
|
```python |
|
import torch |
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
|
|
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') |
|
|
|
|
|
""" |
|
Category includes: ['thời sự ', 'thế giới', 'tài chính kinh doanh', 'đời sống', 'văn hoá', 'giải trí', 'giới trẻ', 'giáo dục','công nghệ', 'sức khoẻ'] |
|
""" |
|
|
|
category = "thời sự" |
|
headline = "Nam thanh niên" # A full headline or only some text |
|
|
|
text = f"<|startoftext|> {category} <|headline|> {headline}" |
|
|
|
tokenizer = AutoTokenizer.from_pretrained("tuanle/VN-News-GPT2") |
|
model= AutoModelForCausalLM.from_pretrained("tuanle/VN-News-GPT2").to(device) |
|
|
|
input_ids = tokenizer.encode(text, return_tensors='pt').to(device) |
|
sample_outputs = model.generate(input_ids, |
|
do_sample=True, |
|
max_length=max_len, |
|
min_length=min_len, |
|
# temperature = .8, |
|
top_k= top_k, |
|
top_p = top_p, |
|
num_beams= num_beams, |
|
early_stopping= True, |
|
no_repeat_ngram_size= 2 , |
|
num_return_sequences= num_return_sequences) |
|
|
|
for i, sample_output in enumerate(sample_outputs): |
|
temp = tokenizer.decode(sample_output.tolist()) |
|
print(f">> Generated text {i+1}\n\n{temp}") |
|
print('\n---') |
|
``` |
|
|
