light_generator / added_tokens.json
orendar's picture
add tokenizer
aef7991
raw
history blame contribute delete
72 Bytes
{"<|startoftext|>": 50258, "<|unknown|>": 50259, "<|endoftext|>": 50257}