Super minimal implementation of GPT-2 in Haskell.
Highly inspired by:
Structure is similar to picoGPT. Code contains:
- Translation of OpenAI's BPE Tokenizer
- Decoder-only transformer
- GPT-2 generation
You can run tests with
cabal build
cabal test
(You'll first need to download GPT-2 weights, tokenizer, and hyperparams into /models)
- Merge changes containing top-k sampling and tensorflow inference for actual text generation
- Nix deps management