End of training

README.md

@@ -0,0 +1,91 @@
+---
+base_model: AIRI-Institute/gena-lm-bert-large-t2t
+tags:
+- generated_from_trainer
+metrics:
+- precision
+- recall
+- accuracy
+model-index:
+- name: gena-lm-bert-large-t2t_ft_BioS73_1kbpHG19_DHSs_H3K27AC
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# gena-lm-bert-large-t2t_ft_BioS73_1kbpHG19_DHSs_H3K27AC
+
+This model is a fine-tuned version of [AIRI-Institute/gena-lm-bert-large-t2t](https://huggingface.co/AIRI-Institute/gena-lm-bert-large-t2t) on the None dataset.
+It achieves the following results on the evaluation set:
+- Loss: 0.6348
+- F1 Score: 0.8584
+- Precision: 0.8916
+- Recall: 0.8275
+- Accuracy: 0.8543
+- Auc: 0.9340
+- Prc: 0.9335
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 1e-05
+- train_batch_size: 8
+- eval_batch_size: 8
+- seed: 42
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- num_epochs: 20
+- mixed_precision_training: Native AMP
+
+### Training results
+
+| Training Loss | Epoch  | Step  | Validation Loss | F1 Score | Precision | Recall | Accuracy | Auc    | Prc    |
+|:-------------:|:------:|:-----:|:---------------:|:--------:|:---------:|:------:|:--------:|:------:|:------:|
+| 0.5295        | 0.1864 | 500   | 0.4614          | 0.8310   | 0.7748    | 0.8959 | 0.8054   | 0.8913 | 0.8893 |
+| 0.438         | 0.3727 | 1000  | 0.4338          | 0.8425   | 0.8408    | 0.8443 | 0.8315   | 0.8998 | 0.8978 |
+| 0.4083        | 0.5591 | 1500  | 0.3931          | 0.8497   | 0.8354    | 0.8645 | 0.8367   | 0.9098 | 0.9056 |
+| 0.4019        | 0.7454 | 2000  | 0.4137          | 0.8525   | 0.8114    | 0.8980 | 0.8341   | 0.9109 | 0.9072 |
+| 0.3802        | 0.9318 | 2500  | 0.4230          | 0.8567   | 0.8471    | 0.8666 | 0.8453   | 0.9165 | 0.9141 |
+| 0.384         | 1.1182 | 3000  | 0.3671          | 0.8640   | 0.8520    | 0.8764 | 0.8528   | 0.9237 | 0.9212 |
+| 0.3603        | 1.3045 | 3500  | 0.4237          | 0.8604   | 0.8193    | 0.9057 | 0.8431   | 0.9195 | 0.9176 |
+| 0.3552        | 1.4909 | 4000  | 0.3575          | 0.8612   | 0.8736    | 0.8492 | 0.8539   | 0.9256 | 0.9210 |
+| 0.3379        | 1.6772 | 4500  | 0.4773          | 0.8607   | 0.8481    | 0.8736 | 0.8490   | 0.9227 | 0.9205 |
+| 0.3469        | 1.8636 | 5000  | 0.4061          | 0.8714   | 0.8143    | 0.9372 | 0.8524   | 0.9291 | 0.9265 |
+| 0.3193        | 2.0499 | 5500  | 0.4850          | 0.8700   | 0.8733    | 0.8666 | 0.8617   | 0.9308 | 0.9291 |
+| 0.3267        | 2.2363 | 6000  | 0.5780          | 0.8727   | 0.8230    | 0.9288 | 0.8554   | 0.9324 | 0.9309 |
+| 0.3239        | 2.4227 | 6500  | 0.3841          | 0.8715   | 0.8386    | 0.9071 | 0.8572   | 0.9286 | 0.9261 |
+| 0.3193        | 2.6090 | 7000  | 0.4107          | 0.8768   | 0.8837    | 0.8701 | 0.8695   | 0.9350 | 0.9314 |
+| 0.336         | 2.7954 | 7500  | 0.4738          | 0.8435   | 0.8995    | 0.7940 | 0.8427   | 0.9338 | 0.9322 |
+| 0.3059        | 2.9817 | 8000  | 0.3902          | 0.8797   | 0.8390    | 0.9246 | 0.8651   | 0.9358 | 0.9323 |
+| 0.2843        | 3.1681 | 8500  | 0.4350          | 0.8777   | 0.8635    | 0.8925 | 0.8673   | 0.9329 | 0.9291 |
+| 0.2812        | 3.3545 | 9000  | 0.6741          | 0.8709   | 0.7963    | 0.9609 | 0.8479   | 0.9334 | 0.9287 |
+| 0.2912        | 3.5408 | 9500  | 0.5464          | 0.8427   | 0.9014    | 0.7912 | 0.8423   | 0.9351 | 0.9350 |
+| 0.2739        | 3.7272 | 10000 | 0.5261          | 0.8832   | 0.8436    | 0.9267 | 0.8692   | 0.9336 | 0.9291 |
+| 0.2778        | 3.9135 | 10500 | 0.5400          | 0.8702   | 0.8817    | 0.8589 | 0.8632   | 0.9334 | 0.9297 |
+| 0.2641        | 4.0999 | 11000 | 0.5587          | 0.8824   | 0.8504    | 0.9169 | 0.8695   | 0.9338 | 0.9292 |
+| 0.2478        | 4.2862 | 11500 | 0.7934          | 0.8188   | 0.9046    | 0.7479 | 0.8233   | 0.9301 | 0.9257 |
+| 0.2214        | 4.4726 | 12000 | 0.5377          | 0.8825   | 0.8687    | 0.8966 | 0.8725   | 0.9331 | 0.9276 |
+| 0.2662        | 4.6590 | 12500 | 0.6348          | 0.8584   | 0.8916    | 0.8275 | 0.8543   | 0.9340 | 0.9335 |
+
+
+### Framework versions
+
+- Transformers 4.42.3
+- Pytorch 2.3.0+cu121
+- Datasets 2.18.0
+- Tokenizers 0.19.0

config.json

@@ -0,0 +1,31 @@
+{
+  "_name_or_path": "AIRI-Institute/gena-lm-bert-large-t2t",
+  "architectures": [
+    "BertForSequenceClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "auto_map": {
+    "AutoModel": "AIRI-Institute/gena-lm-bert-large-t2t--modeling_bert.BertForMaskedLM"
+  },
+  "classifier_dropout": null,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 1024,
+  "initializer_range": 0.02,
+  "intermediate_size": 4096,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 24,
+  "pad_token_id": 3,
+  "position_embedding_type": "absolute",
+  "pre_layer_norm": true,
+  "problem_type": "single_label_classification",
+  "torch_dtype": "float32",
+  "transformers_version": "4.42.3",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 32000
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ff5d18b06daf315d4339a60bc1c5130bf9080076a0f32ff52a8f0ab5a6ad134f
+size 1346688816

special_tokens_map.json

@@ -0,0 +1,37 @@
+{
+  "cls_token": {
+    "content": "[CLS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "[SEP]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,60 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "-",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "[UNK]"
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5e0426b0f8090cb07dad38dc6ef4466519acbc48561ba730003c9a7bc7b13f43
+size 5240