CodeGPT-Multilingual / added_tokens.json
claudios's picture
Upload tokenizer
a7d904b verified
{
"</s>": 50258,
"<EOL>": 50259,
"<pad>": 50260,
"<s>": 50257,
"<|UNKNOWN|>": 50261
}