Llama-PaperSummarization-LoRA

Model Details

This is a LoRA fine-tuned adapter built on meta-llama/Llama-3.2-1B-Instruct. It is designed for scientific paper summarization tasks and leverages Low-Rank Adaptation (LoRA) to enhance model performance efficiently while maintaining a low computational overhead.

Performance comparison

Model ROUGE-1 ROUGE-2 ROUGE-3 ROUGE-L
Llama-3.2-1B-Instruct 36.69 7.47 1.95 19.36
Llama-PaperSummarization-LoRA 41.56 11.31 2.67 21.86

The model was evaluated on a 6K-sample test set using ROUGE scores with beam search (beam size = 4).

How to load

from transformers import LlamaForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = LlamaForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B-Instruct")
peft_model_id = "gabe-zhang/Llama-PaperSummarization-LoRA"
model = PeftModel.from_pretrained(base_model, peft_model_id)
model.merge_and_unload()

Dataset

The model was fine-tuned on the armanc/scientific_papers dataset. Below are the details of the dataset splits:

  • Training Set: 20K samples
  • Validation Set: 6K samples
  • Test Set: 6K samples

LoRA Configuration

  • Trainable Parameters: 850K (~7% of base model parameters)
  • Context Length: 10K tokens
  • Rank: 8
  • Target Modules: Query and Value matrices
  • Optimization Settings:
    • Gradient Accumulation: 4 steps
    • Training Steps: 5K

Training Setup

  • Hardware: NVIDIA RTX A6000 GPU
  • Evaluation Frequency: Every 20 steps
  • Training Duration: 28 hours
  • Training Scripts: gabe-zhang/paper2summary

License

This repository contains a LoRA fine-tuned adapter derived from the Llama 3.2 model.

Attribution

  • The model prominently incorporates Llama 3.2 as its base.
  • "Built with Llama" is displayed as required by the Llama 3.2 Community License.
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for gabe-zhang/Llama-PaperSummarization-LoRA

Adapter
(158)
this model

Dataset used to train gabe-zhang/Llama-PaperSummarization-LoRA