---
library_name: transformers
tags:
- generated_from_trainer
datasets:
- gokulsrinivasagan/processed_book_corpus-ld-100
metrics:
- accuracy
model-index:
- name: bert_tiny_lda_100_v1_book
  results:
  - task:
      name: Masked Language Modeling
      type: fill-mask
    dataset:
      name: gokulsrinivasagan/processed_book_corpus-ld-100
      type: gokulsrinivasagan/processed_book_corpus-ld-100
    metrics:
    - name: Accuracy
      type: accuracy
      value: 0.6765868602733887
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# bert_tiny_lda_100_v1_book

This model is a fine-tuned version of [](https://huggingface.co/) on the gokulsrinivasagan/processed_book_corpus-ld-100 dataset.
It achieves the following results on the evaluation set:
- Loss: 4.9524
- Accuracy: 0.6766

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 160
- eval_batch_size: 160
- seed: 10
- optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 10000
- num_epochs: 25

### Training results

| Training Loss | Epoch   | Step   | Validation Loss | Accuracy |
|:-------------:|:-------:|:------:|:---------------:|:--------:|
| 10.2267       | 0.7025  | 10000  | 10.0665         | 0.1636   |
| 7.2173        | 1.4051  | 20000  | 6.7451          | 0.4823   |
| 6.4475        | 2.1076  | 30000  | 6.0308          | 0.5509   |
| 6.1492        | 2.8102  | 40000  | 5.7522          | 0.5804   |
| 5.9566        | 3.5127  | 50000  | 5.5702          | 0.6009   |
| 5.8259        | 4.2153  | 60000  | 5.4546          | 0.6137   |
| 5.7396        | 4.9178  | 70000  | 5.3672          | 0.6239   |
| 5.6664        | 5.6203  | 80000  | 5.3074          | 0.6312   |
| 5.6155        | 6.3229  | 90000  | 5.2622          | 0.6366   |
| 5.5704        | 7.0254  | 100000 | 5.2177          | 0.6416   |
| 5.5381        | 7.7280  | 110000 | 5.1869          | 0.6460   |
| 5.5072        | 8.4305  | 120000 | 5.1572          | 0.6495   |
| 5.476         | 9.1331  | 130000 | 5.1399          | 0.6520   |
| 5.4586        | 9.8356  | 140000 | 5.1144          | 0.6554   |
| 5.4395        | 10.5381 | 150000 | 5.0980          | 0.6573   |
| 5.4279        | 11.2407 | 160000 | 5.0854          | 0.6588   |
| 5.4084        | 11.9432 | 170000 | 5.0694          | 0.6610   |
| 5.3943        | 12.6458 | 180000 | 5.0544          | 0.6627   |
| 5.3829        | 13.3483 | 190000 | 5.0477          | 0.6636   |
| 5.374         | 14.0509 | 200000 | 5.0361          | 0.6652   |
| 5.3602        | 14.7534 | 210000 | 5.0257          | 0.6666   |
| 5.3506        | 15.4560 | 220000 | 5.0155          | 0.6681   |
| 5.3443        | 16.1585 | 230000 | 5.0103          | 0.6687   |
| 5.3334        | 16.8610 | 240000 | 5.0030          | 0.6697   |
| 5.3252        | 17.5636 | 250000 | 4.9964          | 0.6705   |
| 5.3187        | 18.2661 | 260000 | 4.9904          | 0.6711   |
| 5.3167        | 18.9687 | 270000 | 4.9849          | 0.6723   |
| 5.3068        | 19.6712 | 280000 | 4.9791          | 0.6731   |
| 5.3031        | 20.3738 | 290000 | 4.9740          | 0.6736   |
| 5.2947        | 21.0763 | 300000 | 4.9701          | 0.6742   |
| 5.2931        | 21.7788 | 310000 | 4.9633          | 0.6752   |
| 5.2875        | 22.4814 | 320000 | 4.9602          | 0.6756   |
| 5.2841        | 23.1839 | 330000 | 4.9582          | 0.6758   |
| 5.2815        | 23.8865 | 340000 | 4.9541          | 0.6762   |
| 5.2811        | 24.5890 | 350000 | 4.9512          | 0.6766   |


### Framework versions

- Transformers 4.46.1
- Pytorch 2.2.0+cu121
- Datasets 3.1.0
- Tokenizers 0.20.1