Hugging Face's logo

damgomz
/
ThunBERT_bs16_lr6

Fill-Mask
Transformers
Safetensors
English
albert
pretraining
Inference Endpoints
Model card Files Community
ThunBERT_bs16_lr6
File size: 2,961 Bytes 
2a7098a
736566d
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2a7098a
 
736566d
2a7098a
736566d
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2a7098a
736566d
2a7098a
736566d
 
 
 
2a7098a
736566d
2a7098a
736566d
2a7098a
736566d
2a7098a
736566d
 
 
 
 
 
 
 
 
 
 
 
 
 
2a7098a
736566d
2a7098a
 
 
 
 
 
736566d
 
 
 
65988a6
82a15a2
f401ebd
28baad4
3df9c85
6f80516
dedccf8
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
 
---
language: en
tags:
- fill-mask
kwargs:
  timestamp: '2024-05-11T09:56:27'
  project_name: ThunBERT_bs16_lr6_emissions_tracker
  run_id: 96f76eeb-db87-4131-8576-a7d55fc189c3
  duration: 155341.91502785683
  emissions: 0.1625941105037492
  emissions_rate: 1.046685374482424e-06
  cpu_power: 42.5
  gpu_power: 0.0
  ram_power: 37.5
  cpu_energy: 1.8338952919413605
  gpu_energy: 0
  ram_energy: 1.6181356735105257
  energy_consumed: 3.4520309654518844
  country_name: Switzerland
  country_iso_code: CHE
  region: .nan
  cloud_provider: .nan
  cloud_region: .nan
  os: Linux-5.14.0-70.30.1.el9_0.x86_64-x86_64-with-glibc2.34
  python_version: 3.10.4
  codecarbon_version: 2.3.4
  cpu_count: 4
  cpu_model: Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz
  gpu_count: .nan
  gpu_model: .nan
  longitude: .nan
  latitude: .nan
  ram_total_size: 100
  tracking_mode: machine
  on_cloud: N
  pue: 1.0
---

## Environmental Impact (CODE CARBON DEFAULT)

| Metric                   | Value                           |
|--------------------------|---------------------------------|
| Duration (in seconds)    | 155341.91502785683  |
| Emissions (Co2eq in kg)  | 0.1625941105037492 |
| CPU power (W)            | 42.5  |
| GPU power (W)            | [No GPU]  |
| RAM power (W)            | 37.5  |
| CPU energy (kWh)         | 1.8338952919413605  |
| GPU energy (kWh)         | [No GPU]  |
| RAM energy (kWh)         | 1.6181356735105257  |
| Consumed energy (kWh)    | 3.4520309654518844  |
| Country name             | Switzerland  |
| Cloud provider           | nan  |
| Cloud region             | nan  |
| CPU count                | 4  |
| CPU model                | Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz  |
| GPU count                | nan  |
| GPU model                | nan  |

## Environmental Impact (for one core)

| Metric                   | Value                           |
|--------------------------|---------------------------------|
| CPU energy (kWh)         | 0.29903318642862436  |
| Emissions (Co2eq in kg)  | 0.060842250052577256 |

## Note

15 May 2024

## My Config

| Config                   | Value           |
|--------------------------|-----------------|
| checkpoint               | albert-base-v2  |
| model_name               | ThunBERT_bs16_lr6 |
| sequence_length          | 400  |
| num_epoch                | 6  |
| learning_rate            | 5e-06  |
| batch_size               | 16  |
| weight_decay             | 0.0  |
| warm_up_prop             | 0.0  |
| drop_out_prob            | 0.1 |
| packing_length           | 100 |
| train_test_split         | 0.2 |
| num_steps                | 41105 |

## Training and Testing steps






 
Epoch | Train Loss | Test Loss
---|---|---
| 0.0 | 4.300614 | 5.886359 |
| 0.5 | 3.106338 | 3.011006 |
| 1.0 | 2.976321 | 2.909148 |
| 1.5 | 2.917792 | 2.861870 |
| 2.0 | 2.876170 | 2.839771 |
| 2.5 | 2.848535 | 2.812239 |
| 3.0 | 2.811240 | 2.789157 |
| 3.5 | 2.795877 | 2.775617 |