File size: 3,212 Bytes
d654f89 f370283 d654f89 43fb8c2 f99f2ed f370283 d654f89 546e33f d654f89 a195413 d654f89 8870685 d654f89 484a6eb 47de52e 484a6eb a195413 484a6eb 47de52e 484a6eb 47de52e 484a6eb d654f89 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 |
---
license: apache-2.0
datasets:
- samsum
language:
- en
metrics:
- rouge
library_name: adapter-transformers
model-index:
- name: bart-large-cnn
results:
- task:
name: Summarization
type: summarization
dataset:
name: samsum
type: samsum
split: validation
metrics:
- name: Rouge1
type: rouge
value: 43.115
pipeline_tag: summarization
inference: True
---
# bart-large-cnn-finetuned-samsum-lora
This model is a further fine-tuned version of [facebook/bart-large-cnn](https://huggingface.co/facebook/bart-large-cnn) on the [samsum](https://huggingface.co/datasets/samsum) dataset.
The base model [bart-large-cnn](https://huggingface.co/facebook/bart-large-cnn) is a fine-tuned verstion of BART model on the [CNN Daily Mail](https://huggingface.co/datasets/cnn_dailymail) dataset.
Check out [sooolee/flan-t5-base-cnn-samsum-lora](https://huggingface.co/sooolee/flan-t5-base-cnn-samsum-lora) the model fine-tuned for the same purpose.
## Model description
* This model further finetuned 'bart-large-cnn' on the more conversational samsum dataset.
* Huggingface [PEFT Library](https://github.com/huggingface/peft) LoRA (r = 8) was used to speed up training and reduce the model size.
* Less than 1.2M parameters were trained (0.23% of original bart-large 510M parameters).
* The model checkpoint is less than 5MB.
## Intended uses & limitations
Summarizes transcripts such as YouTube transcripts.
## Training and evaluation data
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0005
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 5
### Training results
- train_loss: 1.28
- rogue1: 43.115465%
- rouge2: 21.563061%
- rougeL: 33.409979%
- rougeLsum: 33.414162%
### How to use
Note 'max_new_tokens=60' is used in the example below to control the summary size. BART model has max generation length = 142 (default) and min generation length = 56.
```python
import torch
from peft import PeftModel, PeftConfig
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
# Load peft config for pre-trained checkpoint etc.
peft_model_id = "sooolee/bart-large-cnn-finetuned-samsum-lora"
config = PeftConfig.from_pretrained(peft_model_id)
# load base LLM model and tokenizer
model = AutoModelForSeq2SeqLM.from_pretrained(config.base_model_name_or_path, device_map='auto')
tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)
# Load the Lora model
model = PeftModel.from_pretrained(model, peft_model_id, device_map='auto')
# Tokenize the text inputs
texts = "<e.g. Transcript>"
inputs = tokenizer(texts, return_tensors="pt", padding=True, ) # truncation=True
# Make inferences
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
with torch.no_grad():
output = self.model.generate(input_ids=inputs["input_ids"].to(device), max_new_tokens=60, do_sample=True, top_p=0.9)
summary = self.tokenizer.batch_decode(output.detach().cpu().numpy(), skip_special_tokens=True)
summary
```
### Framework versions
- Transformers 4.27.2
- Pytorch 1.13.1+cu116
- Datasets 2.9.0
- Tokenizers 0.13.3 |