Mini bpe in Rust Port minbpe to rust as learning process Benchmark Build binary cargo build --release Run tokenizer ./target/release/rbpe --tokenizer {basic, regex} Results On my m1 book, I got: Mode Time (rust) Time (python) Basic 0.4s 5.65s Regex 1.23s 9.01s