Initial commit

README.md

@@ -0,0 +1,173 @@
+---
+language: fr
+license: apache-2.0
+datasets:
+- wikipedia
+---
+
+# FrALBERT Base Cased
+
+Pretrained model on French language using a masked language modeling (MLM) objective. It was introduced in
+[this paper](https://arxiv.org/abs/1909.11942) and first released in
+[this repository](https://github.com/google-research/albert).
+This model, unlike other ALBERT models, is cased: it does make a difference between french and French.
+
+## Model description
+
+FrALBERT is a transformers model pretrained on 16Go of French Wikipedia in a self-supervised fashion. This means it
+was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of
+publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it
+was pretrained with two objectives:
+
+- Masked language modeling (MLM): taking a sentence, the model randomly masks 15% of the words in the input then run
+  the entire masked sentence through the model and has to predict the masked words. This is different from traditional
+  recurrent neural networks (RNNs) that usually see the words one after the other, or from autoregressive models like
+  GPT which internally mask the future tokens. It allows the model to learn a bidirectional representation of the
+  sentence.
+- Sentence Ordering Prediction (SOP): FrALBERT uses a pretraining loss based on predicting the ordering of two consecutive segments of text.
+
+This way, the model learns an inner representation of the English language that can then be used to extract features
+useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard
+classifier using the features produced by the FrALBERT model as inputs.
+
+FrALBERT is particular in that it shares its layers across its Transformer. Therefore, all layers have the same weights. Using repeating layers results in a small memory footprint, however, the computational cost remains similar to a BERT-like architecture with the same number of hidden layers as it has to iterate through the same number of (repeating) layers.
+
+This is the second version of the base model.
+
+This model has the following configuration:
+
+- 12 repeating layers
+- 128 embedding dimension
+- 768 hidden dimension
+- 12 attention heads
+- 11M parameters
+
+## Intended uses & limitations
+
+You can use the raw model for either masked language modeling or next sentence prediction, but it's mostly intended to
+be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?filter=fralbert-base-cased) to look for
+fine-tuned versions on a task that interests you.
+
+Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked)
+to make decisions, such as sequence classification, token classification or question answering. For tasks such as text
+generation you should look at model like GPT2.
+
+### How to use
+
+You can use this model directly with a pipeline for masked language modeling:
+
+```python
+>>> from transformers import pipeline
+>>> unmasker = pipeline('fill-mask', model='cservan/fralbert-base-cased')
+>>> unmasker("Paris est la capitale de la  [MASK] .")
+[
+  {
+    "sequence": "paris est la capitale de la france.",
+    "score": 0.6231236457824707,
+    "token": 3043,
+    "token_str": "france"
+  },
+  {
+    "sequence": "paris est la capitale de la region.",
+    "score": 0.2993471622467041,
+    "token": 10531,
+    "token_str": "region"
+  },
+  {
+    "sequence": "paris est la capitale de la societe.",
+    "score": 0.02028230018913746,
+    "token": 24622,
+    "token_str": "societe"
+  },
+  {
+    "sequence": "paris est la capitale de la bretagne.",
+    "score": 0.012089950032532215,
+    "token": 24987,
+    "token_str": "bretagne"
+  },
+  {
+    "sequence": "paris est la capitale de la chine.",
+    "score": 0.010002839379012585,
+    "token": 14860,
+    "token_str": "chine"
+  }
+]
+```
+
+Here is how to use this model to get the features of a given text in PyTorch:
+
+```python
+from transformers import AlbertTokenizer, AlbertModel
+tokenizer = AlbertTokenizer.from_pretrained('cservan/fralbert-base-cased')
+model = AlbertModel.from_pretrained("cservan/fralbert-base-cased")
+text = "Remplacez-moi par le texte en français que vous souhaitez."
+encoded_input = tokenizer(text, return_tensors='pt')
+output = model(**encoded_input)
+```
+
+and in TensorFlow:
+
+```python
+from transformers import AlbertTokenizer, TFAlbertModel
+tokenizer = AlbertTokenizer.from_pretrained('cservan/fralbert-base-cased')
+model = TFAlbertModel.from_pretrained("cservan/fralbert-base-cased")
+text = "Remplacez-moi par le texte en français que vous souhaitez."
+encoded_input = tokenizer(text, return_tensors='tf')
+output = model(encoded_input)
+```
+
+
+## Training data
+
+The FrALBERT model was pretrained on 4go of [French Wikipedia](https://fr.wikipedia.org/wiki/French_Wikipedia) (excluding lists, tables and
+headers).
+
+## Training procedure
+
+### Preprocessing
+
+The texts are lowercased and tokenized using SentencePiece and a vocabulary size of 32,000. The inputs of the model are
+then of the form:
+
+```
+[CLS] Sentence A [SEP] Sentence B [SEP]
+```
+
+### Training
+
+The FrALBERT procedure follows the BERT setup.
+
+The details of the masking procedure for each sentence are the following:
+- 15% of the tokens are masked.
+- In 80% of the cases, the masked tokens are replaced by `[MASK]`.
+- In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
+- In the 10% remaining cases, the masked tokens are left as is.
+
+## Evaluation results
+
+When fine-tuned on downstream tasks, the ALBERT models achieve the following results:
+
+Slot-filling:
+
+|                | FrALBERT-base | FrALBERT-base-cased
+|----------------|---------------|--------------------
+| MEDIA          | 81.76 (0.59)  | 85.09 (0.14)
+|
+
+
+### BibTeX entry and citation info
+
+```bibtex
+@inproceedings{cattan2021fralbert,
+  author    = {Oralie Cattan and
+               Christophe Servan and
+               Sophie Rosset},
+  booktitle = {Recent Advances in Natural Language Processing, RANLP 2021},
+  title     = {{On the Usability of Transformers-based models for a French Question-Answering task}},
+  year      = {2021},
+  address   = {Online},
+  month     = sep,
+}
+```
+
+Link to the paper: [PDF](https://hal.archives-ouvertes.fr/hal-03336060)

config.json

@@ -0,0 +1,33 @@
+{
+  "architectures": [
+    "AlbertForMaskedLM"
+  ],
+  "attention_probs_dropout_prob": 0,
+  "bos_token_id": 2,
+  "classifier_dropout_prob": 0.1,
+  "down_scale_factor": 1,
+  "embedding_size": 128,
+  "eos_token_id": 3,
+  "eos_token_ids": null,
+  "gap_size": 0,
+  "hidden_act": "gelu_new",
+  "hidden_dropout_prob": 0,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "inner_group_num": 1,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "albert",
+  "net_structure_type": 0,
+  "num_attention_heads": 12,
+  "num_hidden_groups": 1,
+  "num_hidden_layers": 12,
+  "num_memory_blocks": 0,
+  "output_past": true,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "transformers_version": "4.24.0",
+  "type_vocab_size": 2,
+  "vocab_size": 32000
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:86789a8f6e22063bce5c8c944db629b7c1c5544761d4a6ecf0bc9b8540a653bc
+size 64676287

special_tokens_map.json

@@ -0,0 +1,3 @@
+{"bos_token": "[CLS]", "eos_token": "[SEP]", "unk_token": "<unk>", "sep_token": "[SEP]", "pad_token": "<pad>", "cls_token": "[CLS]", "mask_token": "[MASK]"}
+
+

spiece.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f84b84d236af93fa229f8c3e400c02b31c8532e61b1b927f609a8b470d4afb71
+size 737469

tokenizer_config.json

@@ -0,0 +1,2 @@
+{"keep_accents": true, "do_lower_case": false}
+