#
sentencepiece
Here are 3 public repositories matching this topic...
Fast and versatile tokenizer for language models with BPE, Unigram and WordPiece tokenization. Compatible with SentencePiece, Tokenizers, Tiktoken and more.
-
Updated
Aug 7, 2024 - Rust
SentencePiece model parser generated from the SentencePiece protobuf definition.
-
Updated
Jul 16, 2024 - Rust
Improve this page
Add a description, image, and links to the sentencepiece topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the sentencepiece topic, visit your repo's landing page and select "manage topics."