Update README.md

README.md

@@ -48,13 +48,14 @@ tags:
 <a name="model-introduction"></a><br>
 # 1. Model Introduction
 
-- Orion-MOE8x7B-Base Large Language Model(LLM) is a pretrained generative Sparse Mixture of Experts, trained from scratch by OrionStarAI.  The base model is trained on multilingual corpus, including Chinese, English, Japanese, Korean, etc, and it exhibits superior performance in these languages.
+- Orion-MOE8x7B is a pretrained foundation large language model with a sparse Mixture of Experts (MoE) architecture. The model is trained from scratch on a multilingual corpus comprising approximately 5 trillion tokens, including launguages such as Chinese, English, Japanese, Korean, and more.
 
-- The Orion-MOE8x7B series models exhibit the following features
-  - The model demonstrates excellent performance in comprehensive evaluations compared to other base models of the same parameter scale.
-  - It has strong multilingual capabilities, significantly leading in Japanese and Korean test sets, and also performing comprehensively better in Arabic, German, French, and Spanish test sets.
-- Model Hyper-Parameters
-  - The architecture of the OrionMOE 8x7B models closely resembles that of Mixtral 8x7B, with specific details shown in the table below.
+- Key Features of Orion-MoE8x7B
+  - The model demonstrates exceptional performance in comprehensive evaluations compared to other models of the same parameter scale.
+  - The model excels in multilingual benchmarks, significantly outperforming in Japanese and Korean test sets, and also performing strong results in Arabic, German, French, and Spanish evaluations.
+  - Leveraging its sparse MoE structure, the model achieves faster inference speeds compared to dense models of similar scale.
+
+- Model Architecture 
 
     |Configuration      |OrionMOE 8x7B|
     |-------------------|-------------|
@@ -70,12 +71,13 @@ tags:
     |seq_len            | 8192        |
     |Vocabulary Size    | 1136664     |
 
-- Model pretrain hyper-parameters
+- Training hyper-parameters
   - We use the AdamW optimizer with hyperparameters set to 𝛽1 = 0.9, 𝛽2 = 0.95, and a weight decay of 0.1.
   - Training begins with a learning rate warm-up phase over 2000 iterations, where the learning rate is linearly increased to a peak of 3e-4. Afterward, a cosine schedule is applied to gradually reduce the learning rate to 3e-5 over the course of training.
   - The model is trained using BF16/FP32 mixed precision, with a batch size of 2600, processing approximately 22 million tokens per step.
-- Model pretrain data distribution
-  - The training dataset is primarily composed of English, Chinese, and other languages, accounting for 50%, 25%, and 12% of the data, respectively. Additionally, code makes up 9%, while mathematical text accounts for 4%. The distribution by topics is detailed in the table below.
+
+- Data Distribution
+  - The training dataset is primarily composed of English, Chinese, which together account for over 75% of the total data. The remaining dataset includes other languages, programming code, mathematical data, etc. A detailed breakdown of the topic distribution is provided in the table below.
 <div align="center">
   <img src="./assets/imgs/data_src_dist.png" alt="logo" width="50%" />
 </div>