Releases: niieani/gpt-tokenizer
Releases · niieani/gpt-tokenizer
2.2.1
2.2.1 (2024-07-18)
Bug Fixes
- add files for other models (c21d498), closes #19
- regenerate o200k encoding from tiktoken file (c7ba009)
2.2.0
2.2.0 (2024-07-17)
Bug Fixes
- add correct encoding for o200k_base (137e07b)
- add gpt-4o on readme as supported model (27b4e20)
- update readme (0b33e1e)
Features
- add o200k_base test plans (44ce38e)
- added o200k_base to encodings and configured it's specialTokens (2a9da2b)
2.1.2
2.1.2 (2023-10-07)
Bug Fixes
- bind encodeChat and encodeChatGenerator (86c270c), closes #15
2.1.1
2.1.1 (2023-06-01)
Bug Fixes
- missing default exports due to limitations of ESM (2a55474), closes #11
2.1.0
2.1.0 (2023-06-01)
Bug Fixes
- correct excluded path (71af9d3)
- workaround for webpack not exposing the default export in UMD correctly (84887b4), closes #12
Features
2.0.0
2.0.0 (2023-05-24)
Features
- complete rewrite to support different models (eedd944), closes #5 #6
- more tests and better README (e660a25)
BREAKING CHANGES
- default encoder is now GPT3.5 / GPT4
2.0.0-beta.2
Features
- more tests and better README (e660a25)
2.0.0-beta.1
Features
- complete rewrite to support different models (eedd944), closes #5 #6
BREAKING CHANGES
- default encoder is now GPT3.5 / GPT4
1.0.5
1.0.5 (2023-05-16)
Bug Fixes
1.0.4
1.0.4 (2023-05-01)
Bug Fixes
- eslint and jest complaints (8b1cb01)
- importing/requiring under certain setups was not working (8b71131), closes #4