gpt2-chess-uci-hooked / tokenizer_config.json
Austin Davis
Upload tokenizer
04a3e2b
raw
history blame
232 Bytes
{
"add_bos_token": true,
"bos_token": "<s>",
"clean_up_tokenization_spaces": true,
"model_max_length": 1000000000000000019884624838656,
"pad_token": "<pad>",
"tokenizer_class": "ChessTokenizer",
"unk_token": "<pad>"
}