Add bos and eos token to tokenizer config.
#1
by
andrejonasson
- opened
https://huggingface.co/meta-llama/Llama-3.2-1B/blob/main/tokenizer_config.json
Update to be similar to llama tokenizer config.
https://huggingface.co/meta-llama/Llama-3.2-1B/blob/main/tokenizer_config.json
Update to be similar to llama tokenizer config.