Hang Le
commited on
Commit
·
0eccb33
1
Parent(s):
b9a249e
Fix formatting
Browse files
README.md
CHANGED
@@ -3,13 +3,22 @@ language: fr
|
|
3 |
license: mit
|
4 |
datasets:
|
5 |
- flaubertmetrics:
|
6 |
-
-
|
|
|
7 |
- bert
|
8 |
- language-model
|
9 |
-
- flaubert
|
|
|
|
|
|
|
|
|
|
|
|
|
10 |
**FlauBERT** is a French BERT trained on a very large and heterogeneous French corpus. Models of different sizes are trained using the new CNRS (French National Centre for Scientific Research) [Jean Zay](http://www.idris.fr/eng/jean-zay/ ) supercomputer.
|
11 |
|
12 |
-
Along with FlauBERT comes [**FLUE**](https://github.com/getalp/Flaubert/tree/master/flue): an evaluation setup for French NLP systems similar to the popular GLUE benchmark. The goal is to enable further reproducible experiments in the future and to share models and progress on the French language.For more details please refer to the [official website](https://github.com/getalp/Flaubert)
|
|
|
|
|
13 |
|
14 |
| Model name | Number of layers | Attention Heads | Embedding Dimension | Total Parameters |
|
15 |
| :------: | :---: | :---: | :---: | :---: |
|
@@ -18,7 +27,9 @@ Along with FlauBERT comes [**FLUE**](https://github.com/getalp/Flaubert/tree/mas
|
|
18 |
| `flaubert-base-cased` | 12 | 12 | 768 | 138 M |
|
19 |
| `flaubert-large-cased` | 24 | 16 | 1024 | 373 M |
|
20 |
|
21 |
-
Note
|
|
|
|
|
22 |
|
23 |
```python
|
24 |
import torch
|
@@ -50,7 +61,10 @@ of the following values:
|
|
50 |
```
|
51 |
['flaubert-small-cased', 'flaubert-base-uncased', 'flaubert-base-cased', 'flaubert-large-cased']
|
52 |
```
|
|
|
|
|
53 |
## References
|
|
|
54 |
If you use FlauBERT or the FLUE Benchmark for your scientific publication, or if you find the resources in this repository useful, please cite one of the following papers:
|
55 |
|
56 |
[LREC paper](http://www.lrec-conf.org/proceedings/lrec2020/pdf/2020.lrec-1.302.pdf)
|
|
|
3 |
license: mit
|
4 |
datasets:
|
5 |
- flaubertmetrics:
|
6 |
+
- flue
|
7 |
+
tags:
|
8 |
- bert
|
9 |
- language-model
|
10 |
+
- flaubert
|
11 |
+
- flue
|
12 |
+
- french
|
13 |
+
---
|
14 |
+
|
15 |
+
# FlauBERT: Unsupervised Language Model Pre-training for French
|
16 |
+
|
17 |
**FlauBERT** is a French BERT trained on a very large and heterogeneous French corpus. Models of different sizes are trained using the new CNRS (French National Centre for Scientific Research) [Jean Zay](http://www.idris.fr/eng/jean-zay/ ) supercomputer.
|
18 |
|
19 |
+
Along with FlauBERT comes [**FLUE**](https://github.com/getalp/Flaubert/tree/master/flue): an evaluation setup for French NLP systems similar to the popular GLUE benchmark. The goal is to enable further reproducible experiments in the future and to share models and progress on the French language.For more details please refer to the [official website](https://github.com/getalp/Flaubert).
|
20 |
+
|
21 |
+
## FlauBERT models
|
22 |
|
23 |
| Model name | Number of layers | Attention Heads | Embedding Dimension | Total Parameters |
|
24 |
| :------: | :---: | :---: | :---: | :---: |
|
|
|
27 |
| `flaubert-base-cased` | 12 | 12 | 768 | 138 M |
|
28 |
| `flaubert-large-cased` | 24 | 16 | 1024 | 373 M |
|
29 |
|
30 |
+
**Note:** `flaubert-small-cased` is partially trained so performance is not guaranteed. Consider using it for debugging purpose only.
|
31 |
+
|
32 |
+
## Using FlauBERT with Hugging Face's Transformers
|
33 |
|
34 |
```python
|
35 |
import torch
|
|
|
61 |
```
|
62 |
['flaubert-small-cased', 'flaubert-base-uncased', 'flaubert-base-cased', 'flaubert-large-cased']
|
63 |
```
|
64 |
+
|
65 |
+
|
66 |
## References
|
67 |
+
|
68 |
If you use FlauBERT or the FLUE Benchmark for your scientific publication, or if you find the resources in this repository useful, please cite one of the following papers:
|
69 |
|
70 |
[LREC paper](http://www.lrec-conf.org/proceedings/lrec2020/pdf/2020.lrec-1.302.pdf)
|