jmaczan / bpe.c Public

Notifications You must be signed in to change notification settings
Fork 0
Star 1

Byte-Pair Encoding tokenizer for training large language models on huge datasets. I don't know C, so most of the code comes from AI :D I hope to learn by rewriting it and making changes, fixes etc

GPL-3.0 license

1 star 0 forks Branches Tags Activity

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 4 Commits
.vscode		.vscode
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
bpe.c		bpe.c

Repository files navigation

bpe.c

Byte-Pair Encoding tokenizer for training large language models on huge datasets. I don't know C yet, so most of the code comes from AI :D

I hope to learn C by rewriting it and making changes, fixes etc

About

Byte-Pair Encoding tokenizer for training large language models on huge datasets. I don't know C, so most of the code comes from AI :D I hope to learn by rewriting it and making changes, fixes etc

c tokenizer clang bpe llm bpe-tokenizer

GPL-3.0 license

Report repository

Releases

No releases published

Packages

No packages published

Languages

C 100.0%