Hugging Face's logo

nvidia
/
nemo-megatron-gpt-1.3B

Text2Text Generation
NeMo
PyTorch
English
causal-lm
Model card Files Community
1
nemo-megatron-gpt-1.3B / README.md
okuchaiev's picture
okuchaiev
Update README.md
3edf4fd
preview code
|
raw
history blame
3.04 kB
metadata
language:
  - en
library_name: nemo
datasets:
  - the_pile
tags:
  - text generation
  - pytorch
  - causal-lm
license: cc-by-4.0

Megatron-GPT 1.3B

|Model architecture|Model size|Language

Model Description

Megatron-GPT 1.3B is a transformer-based language model. GPT refers to a class of transformer decoder-only models similar to GPT-2 and 3 while 1.3B refers to the total trainable parameter count (1.3 Billion) [1, 2].

This model was trained with NeMo Megatron.

Getting started

Step 1: Install NeMo and dependencies

You will need to install NVIDIA Apex and NeMo. 

git clone https://github.com/ericharper/apex.git
cd apex
git checkout nm_v1.11.0
pip install -v --disable-pip-version-check --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" --global-option="--fast_layer_norm" --global-option="--distributed_adam" --global-option="--deprecated_fused_adam" ./

pip install nemo_toolkit['nlp']==1.11.0

Alternatively, you can use NeMo Megatron training docker container with all dependencies pre-installed.

Step 2: Launch eval server

Note. The model has been trained with Tensor Parallelism (TP) of 1 and Pipeline Parallelism (PP) of 1 and should fit on a single NVIDIA GPU.

git clone https://github.com/NVIDIA/NeMo.git 
cd NeMo/examples/nlp/language_modeling
git checkout v1.11.0
python megatron_gpt_eval.py gpt_model_file=nemo_gpt5B_fp16.nemo server=True tensor_model_parallel_size=1 trainer.devices=1

Training Data

The model was trained on "The Piles" dataset prepared by Eleuther.AI.

Evaluation results

Zero-shot performance.

ARC-Challenge ARC-Easy RACE-middle RACE-high Winogrande RTE BoolQA HellaSwag PiQA
0.3012 0.4596 0.459 0.3811 0.5343 0.5451 0.5979 0.4442 0.6834

References

[1] Improving Language Understanding by Generative Pre-Training

[2] Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

[3] NVIDIA NeMo Toolkit

Licence

License to use this model is covered by the CC-BY-4.0. By downloading the public and release version of the model, you accept the terms and conditions of the CC-BY-4.0 license.