eng-first-tokenizer / added_tokens.json
VaidikML0508's picture
Upload tokenizer
d887831 verified
raw
history blame contribute delete
220 Bytes
{
"</s>": 8001,
"<mask>": 8004,
"<pad>": 8003,
"<s>": 8000,
"<unk>": 8002,
"<|assistant|>": 8008,
"<|context|>": 8010,
"<|endoftext|>": 8005,
"<|end|>": 8007,
"<|system|>": 8009,
"<|user|>": 8006
}