Llama-PaperSummarization-LoRA
Model Details
This is a LoRA fine-tuned adapter built on meta-llama/Llama-3.2-1B-Instruct. It is designed for scientific paper summarization tasks and leverages Low-Rank Adaptation (LoRA) to enhance model performance efficiently while maintaining a low computational overhead.
Performance comparison
Model | ROUGE-1 | ROUGE-2 | ROUGE-3 | ROUGE-L |
---|---|---|---|---|
Llama-3.2-1B-Instruct | 36.69 | 7.47 | 1.95 | 19.36 |
Llama-PaperSummarization-LoRA | 41.56 | 11.31 | 2.67 | 21.86 |
The model was evaluated on a 6K-sample test set using ROUGE scores with beam search (beam size = 4).
How to load
from transformers import LlamaForCausalLM, AutoTokenizer
from peft import PeftModel
base_model = LlamaForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B-Instruct")
peft_model_id = "gabe-zhang/Llama-PaperSummarization-LoRA"
model = PeftModel.from_pretrained(base_model, peft_model_id)
model.merge_and_unload()
Dataset
The model was fine-tuned on the armanc/scientific_papers dataset. Below are the details of the dataset splits:
- Training Set: 20K samples
- Validation Set: 6K samples
- Test Set: 6K samples
LoRA Configuration
- Trainable Parameters: 850K (~7% of base model parameters)
- Context Length: 10K tokens
- Rank: 8
- Target Modules: Query and Value matrices
- Optimization Settings:
- Gradient Accumulation: 4 steps
- Training Steps: 5K
Training Setup
- Hardware: NVIDIA RTX A6000 GPU
- Evaluation Frequency: Every 20 steps
- Training Duration: 28 hours
- Training Scripts: gabe-zhang/paper2summary
License
This repository contains a LoRA fine-tuned adapter derived from the Llama 3.2 model.
- Llama 3.2 Materials: Governed by the Llama 3.2 Community License.
- All other content: Licensed under the MIT License.
Attribution
- The model prominently incorporates Llama 3.2 as its base.
- "Built with Llama" is displayed as required by the Llama 3.2 Community License.
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.
Model tree for gabe-zhang/Llama-PaperSummarization-LoRA
Base model
meta-llama/Llama-3.2-1B-Instruct