Update tokenization_indictrans.py

#3
by VarunGumma - opened

Fix tokenization as per new HF Tokenizer

prajdabre changed pull request status to merged

Sign up or log in to comment