|
--- |
|
language: |
|
- ln |
|
- en |
|
- fr |
|
library_name: peft |
|
tags: |
|
- trl |
|
- sft |
|
- generated_from_trainer |
|
base_model: CohereForAI/aya-23-8b |
|
datasets: |
|
- masakhane/afrimmlu |
|
model-index: |
|
- name: aya-23-8b-afrimmlu-lin |
|
results: [] |
|
--- |
|
|
|
<!-- This model card has been generated automatically according to the information the Trainer had access to. You |
|
should probably proofread and complete it, then remove this comment. --> |
|
|
|
# Aya-23-8b Afrimmlu Lingala |
|
|
|
This model is a fine-tuned version of [CohereForAI/aya-23-8b](https://huggingface.co/CohereForAI/aya-23-8b) on [Masakhane/afrimmlu](https://huggingface.co/datasets/masakhane/afrimmlu/). |
|
|
|
## Model description |
|
|
|
More information needed |
|
|
|
## Intended uses & limitations |
|
|
|
More information needed |
|
|
|
## Training and evaluation data |
|
|
|
More information needed |
|
|
|
## Training procedure |
|
|
|
## Prompt Formating |
|
```py |
|
def formatting_prompts_func(example): |
|
output_texts = [] |
|
for i in range(len(example['choices'])): |
|
text = f"<|START_OF_TURN_TOKEN|><|USER_TOKEN|>Question : {example['question'][i]}, Choices : {example['choices'][i]}<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>{example['answer'][i]}" |
|
output_texts.append(text) |
|
return output_texts |
|
``` |
|
|
|
## Model Architecture |
|
|
|
```txt |
|
PeftModelForCausalLM( |
|
(base_model): LoraModel( |
|
(model): CohereForCausalLM( |
|
(model): CohereModel( |
|
(embed_tokens): Embedding(256000, 4096, padding_idx=0) |
|
(layers): ModuleList( |
|
(0-31): 32 x CohereDecoderLayer( |
|
(self_attn): CohereAttention( |
|
(q_proj): lora.Linear4bit( |
|
(base_layer): Linear4bit(in_features=4096, out_features=4096, bias=False) |
|
(lora_dropout): ModuleDict( |
|
(default): Identity() |
|
) |
|
(lora_A): ModuleDict( |
|
(default): Linear(in_features=4096, out_features=32, bias=False) |
|
) |
|
(lora_B): ModuleDict( |
|
(default): Linear(in_features=32, out_features=4096, bias=False) |
|
) |
|
(lora_embedding_A): ParameterDict() |
|
(lora_embedding_B): ParameterDict() |
|
) |
|
(k_proj): lora.Linear4bit( |
|
(base_layer): Linear4bit(in_features=4096, out_features=1024, bias=False) |
|
(lora_dropout): ModuleDict( |
|
(default): Identity() |
|
) |
|
(lora_A): ModuleDict( |
|
(default): Linear(in_features=4096, out_features=32, bias=False) |
|
) |
|
(lora_B): ModuleDict( |
|
(default): Linear(in_features=32, out_features=1024, bias=False) |
|
) |
|
(lora_embedding_A): ParameterDict() |
|
(lora_embedding_B): ParameterDict() |
|
) |
|
(v_proj): lora.Linear4bit( |
|
(base_layer): Linear4bit(in_features=4096, out_features=1024, bias=False) |
|
(lora_dropout): ModuleDict( |
|
(default): Identity() |
|
) |
|
(lora_A): ModuleDict( |
|
(default): Linear(in_features=4096, out_features=32, bias=False) |
|
) |
|
(lora_B): ModuleDict( |
|
(default): Linear(in_features=32, out_features=1024, bias=False) |
|
) |
|
(lora_embedding_A): ParameterDict() |
|
(lora_embedding_B): ParameterDict() |
|
) |
|
(o_proj): lora.Linear4bit( |
|
(base_layer): Linear4bit(in_features=4096, out_features=4096, bias=False) |
|
(lora_dropout): ModuleDict( |
|
(default): Identity() |
|
) |
|
(lora_A): ModuleDict( |
|
(default): Linear(in_features=4096, out_features=32, bias=False) |
|
) |
|
(lora_B): ModuleDict( |
|
(default): Linear(in_features=32, out_features=4096, bias=False) |
|
) |
|
(lora_embedding_A): ParameterDict() |
|
(lora_embedding_B): ParameterDict() |
|
) |
|
(rotary_emb): CohereRotaryEmbedding() |
|
) |
|
(mlp): CohereMLP( |
|
(gate_proj): Linear4bit(in_features=4096, out_features=14336, bias=False) |
|
(up_proj): Linear4bit(in_features=4096, out_features=14336, bias=False) |
|
(down_proj): Linear4bit(in_features=14336, out_features=4096, bias=False) |
|
(act_fn): SiLU() |
|
) |
|
(input_layernorm): CohereLayerNorm() |
|
) |
|
) |
|
(norm): CohereLayerNorm() |
|
) |
|
(lm_head): Linear(in_features=4096, out_features=256000, bias=False) |
|
) |
|
) |
|
) |
|
``` |
|
|
|
### Training hyperparameters |
|
|
|
The following hyperparameters were used during training: |
|
- learning_rate: 0.0002 |
|
- train_batch_size: 2 |
|
- eval_batch_size: 8 |
|
- seed: 42 |
|
- gradient_accumulation_steps: 16 |
|
- total_train_batch_size: 32 |
|
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 |
|
- lr_scheduler_type: constant |
|
- lr_scheduler_warmup_ratio: 0.05 |
|
- num_epochs: 20 |
|
|
|
### Training results |
|
|
|
|
|
## Inferennce |
|
|
|
```py |
|
quantization_config = None |
|
if QUANTIZE_4BIT: |
|
quantization_config = BitsAndBytesConfig( |
|
load_in_4bit=True, |
|
bnb_4bit_quant_type="nf4", |
|
bnb_4bit_use_double_quant=True, |
|
bnb_4bit_compute_dtype=torch.bfloat16, |
|
) |
|
|
|
attn_implementation = None |
|
if USE_FLASH_ATTENTION: |
|
attn_implementation="flash_attention_2" |
|
|
|
loaded_model = AutoModelForCausalLM.from_pretrained( |
|
BASE_MODEL_NAME, |
|
quantization_config=quantization_config, |
|
attn_implementation=attn_implementation, |
|
torch_dtype=torch.bfloat16, |
|
device_map="auto", |
|
) |
|
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL_NAME) |
|
loaded_model.load_adapter("aya-23-8b-afrimmlu-lin") |
|
|
|
|
|
prompts = [ |
|
"""Question : Kati ya kondima mibale elandi, oyo wapi ezali nyoso mibale ya solo (na 2019) ? |
|
Choices : ['Bato bazali na mposa ya kozala optimiste mpo na mikolo ekoya ya bomoi na bango na mpe na makambo ekoya ya ekolo na bango to mokili.', 'Bato bazali na mposa ya kozala optimiste mpo na mikolo ekoya ya bomoi na bango moko, kasi pessimiste na mikolo ekoya ya Ekolo na bango to mokili.', 'Bato bazali na mposa ya kozala pessimiste mpo na mikolo ekoya ya bomoi na bango, kasi optimiste na mikolo ekoya ya ekolo na bango to mokili.', 'Bato bazali na mposa ya kozala pessimiste mpo na mikolo ekoya ya bomoi na bango na mpe mikolo ekoya ya ekolo na bango to mokili.'] |
|
""" |
|
] |
|
|
|
generations = generate_aya_23(prompts, loaded_model) |
|
|
|
for p, g in zip(prompts, generations): |
|
print( |
|
"PROMPT", p ,"RESPONSE", g, "\n", sep="\n" |
|
) |
|
|
|
|
|
``` |
|
|
|
### Framework versions |
|
|
|
- PEFT 0.11.1 |
|
- Transformers 4.41.2 |
|
- Pytorch 2.1.0+cu118 |
|
- Datasets 2.19.2 |
|
- Tokenizers 0.19.1 |
