alexandra-barker/roberta-lora-gender-bias

README.md

@@ -15,8 +15,6 @@ should probably proofread and complete it, then remove this comment. -->
 # lora_finetuned_roberta_mlm
 
 This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on an unknown dataset.
-It achieves the following results on the evaluation set:
-- Loss: nan
 
 ## Model description
 
@@ -37,7 +35,7 @@ More information needed
 The following hyperparameters were used during training:
 - learning_rate: 1e-05
 - train_batch_size: 1
-- eval_batch_size: 1
+- eval_batch_size: 8
 - seed: 42
 - optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
 - lr_scheduler_type: linear
@@ -45,10 +43,6 @@ The following hyperparameters were used during training:
 
 ### Training results
 
-| Training Loss | Epoch | Step | Validation Loss |
-|:-------------:|:-----:|:----:|:---------------:|
-| 0.0           | 1.0   | 3000 | nan             |
-| 0.0           | 2.0   | 6000 | nan             |
 
 
 ### Framework versions

adapter_config.json

@@ -14,20 +14,16 @@
   "lora_dropout": 0.1,
   "megatron_config": null,
   "megatron_core": "megatron.core",
-  "modules_to_save": [
-    "LayerNorm",
-    "classifier"
-  ],
+  "modules_to_save": null,
   "peft_type": "LORA",
-  "r": 8,
+  "r": 16,
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
     "query",
-    "key",
     "value"
   ],
-  "task_type": "CAUSAL_LM",
+  "task_type": "MASKED_LM",
   "use_dora": false,
   "use_rslora": false
 }

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b154cce916dda8dd7b321018a1584074450bf7f6df37fef4beabc9202d6f67d8
-size 1863152
+oid sha256:1f22e0dd63c6bda6085388b5ba866b9560dae8f09b288956e484ba86be54723c
+size 2366208

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0feea9722bee7f02cba0b683307d7628a78b6b5feabe05f0b68c58d333baa825
+oid sha256:32d31019e9d9f9e57c8e4189d50cdb106bcfc544af8bbca5710f3488e974f1cc
 size 5304