README.md

Mini bpe in Rust

Port minbpe to rust as learning process

Benchmark

Build binary

cargo build --release

Run tokenizer

./target/release/rbpe --tokenizer {basic, regex}

Results

On my m1 book, I got:

Mode	Time (rust)	Time (python)
Basic	0.4s	5.65s
Regex	1.23s	9.01s