Hugging Face's logo

circulus
/
on-bart-vi2ko-v1

Safetensors
bart
Model card Files Community
rippertnt commited on
Commit
f4637f9
·
verified ·
1 Parent(s): 54c47dd

Upload tokenizer

Browse files
Files changed (4) hide show
  1. merges.txt +0 -0
  2. tokenizer.json +0 -0
  3. tokenizer_config.json +1 -0
  4. vocab.json +0 -0
merges.txt CHANGED
The diff for this file is too large to render. See raw diff
 
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json CHANGED
@@ -52,5 +52,6 @@
52
  "pad_token": "<pad>",
53
  "sep_token": "</s>",
54
  "tokenizer_class": "BartTokenizer",
 
55
  "unk_token": "<unk>"
56
  }
 
52
  "pad_token": "<pad>",
53
  "sep_token": "</s>",
54
  "tokenizer_class": "BartTokenizer",
55
+ "trim_offsets": true,
56
  "unk_token": "<unk>"
57
  }
vocab.json CHANGED
The diff for this file is too large to render. See raw diff