fahadqazi
/

Sindhi-BPE-Tokenizer

Inference Endpoints

Model card Files Files and versions Community

fahadqazi commited on Nov 14, 2024

Commit

d9a951f

·

verified ·

1 Parent(s): f3eac7c

Update README.md

Files changed (1) hide show

README.md +2 -2

README.md CHANGED Viewed

@@ -32,8 +32,8 @@ tokenizer = Tokenizer.from_pretrained("fahadqazi/Sindhi-BPE-Tokenizer")
 encoded = tokenizer.encode("ڪهڙا حال آهن")
 decoded = tokenizer.decode(encoded.ids)
-print("Encoded tokens: ", encoded.tokens)
-print("Decoded text: ", decoded)
 ```

 encoded = tokenizer.encode("ڪهڙا حال آهن")
 decoded = tokenizer.decode(encoded.ids)
+print("Encoded tokens: ", encoded.tokens) # output: ['Úª', 'Ùĩ', 'ÚĻØ§', 'ĠØŃØ§ÙĦ', 'ĠØ¢ÙĩÙĨ']
+print("Decoded text: ", decoded) # output: ڪهڙا حال آهن
 ```