---
license: apache-2.0
base_model: Geotrend/distilbert-base-nl-cased
model-index:
- name: distilbert-base-nl-cased-finetuned-squad
  results: []
language:
- nl
pipeline_tag: question-answering
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# distilbert-base-nl-cased-finetuned-squad

This model is a fine-tuned version of [Geotrend/distilbert-base-nl-cased](https://huggingface.co/Geotrend/distilbert-base-nl-cased) on the [Dutch Squad V2 Dataset](https://gitlab.com/niels.rouws/dutch-squad-v2.0) dataset [1], specifically tailored for the Question Answer task.
It achieves the following results on the evaluation set:
- Loss: 1.2834

## Model description

The base model, distilbert-base-nl-cased, is a smaller version of distilbert-base-multilingual-cased, designed to handle a custom number of languages (only Dutch in this case) while preserving the original model's accuracy. It is based on the principles outlined in the paper "Load What You Need: Smaller Versions of Multilingual BERT" by Abdaoui, Pradel, and Sigel (2020) [2].

## Intended uses & limitations

This fine-tuned model is optimized for Dutch Question Answering tasks. While it may perform well on similar tasks in other languages, its primary strength lies in extracting answers from Dutch-language contexts. Users are encouraged to consider the model's specific training focus when applying it to different language or task scenarios.

## Training and evaluation data

The model was trained on the Dutch Squad V2.0 Dataset, a machine-translated version of the original SQuAD v2.0 dataset. The statistics for both datasets are as follows:

### Statistics
```
|                         | SQuAD v2.0       | Dutch SQuAD v2.0 |
|-------------------------|------------------|------------------|
| **Train**               |                  |                  |
| Total examples          |   130,319        |   95,054         |
| Positive examples       |   86,821         |   53,376         |
| Negative examples       |   43,498         |   41,768         |
| **Development**         |                  |                  |
| Total examples          |   11,873         |   9,294          |
| Positive examples       |   5,928          |   3,588          |
| Negative examples       |   5,945          |   5,706          |
```


## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3

### Training results
```
| Training Loss | Epoch | Step  | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| 1.4685        | 1.0   | 6229  | 1.2709          |
| 1.1882        | 2.0   | 12458 | 1.1931          |
| 0.9488        | 3.0   | 18687 | 1.2834          |
```

### Raw Results
```python
{
'exact': 59.479233914353344,
 'f1': 62.56163022484813,
 'total': 9294,
 'HasAns_exact': 38.405797101449274,
 'HasAns_f1': 46.390131357228995,
 'HasAns_total': 3588,
 'NoAns_exact': 72.73045916579039,
 'NoAns_f1': 72.73045916579039,
 'NoAns_total': 5706,
 'best_exact': 61.58812136862492,
 'best_exact_thresh': 0.0,
 'best_f1': 63.337535221120724,
 'best_f1_thresh': 0.0
}
```

## Model Usage

To use this model, you can follow the example below:

```python
from transformers import pipeline

qa_pipeline = pipeline(
    "question-answering",
    model="tclungu/distilbert-base-nl-cased-finetuned-squad",
    tokenizer="tclungu/distilbert-base-nl-cased-finetuned-squad"
)

qa_pipeline({
    'context': "Amsterdam is de hoofdstad en de dichtstbevolkte stad van Nederland.",
    'question': "Wat is de hoofdstad van Nederland?"})

```

### Output
```python
{'score': 0.9984413385391235, 'start': 0, 'end': 9, 'answer': 'Amsterdam'}
```

## Framework versions

- Transformers 4.33.3
- Pytorch 2.0.1
- Datasets 2.14.5
- Tokenizers 0.13.3

### References
```
[1] Rouws, N. J., Vakulenko, S., & Katrenko, S. (2022). Dutch squad and ensemble learning for question answering from labour agreements. In Artificial Intelligence and Machine Learning: 33rd Benelux Conference on Artificial Intelligence, BNAIC/Benelearn 2021, Esch-sur-Alzette, Luxembourg, November 10–12, 2021, Revised Selected Papers 33 (pp. 155-169). Springer International Publishing.
[2] Abdaoui, A., Pradel, C., & Sigel, G. (2020). Load what you need: Smaller versions of multilingual bert. arXiv preprint arXiv:2010.05609.
```