Skip to content

Releases: niieani/gpt-tokenizer

2.2.1

18 Jul 02:01
3d01a89
Compare
Choose a tag to compare

2.2.1 (2024-07-18)

Bug Fixes

  • add files for other models (c21d498), closes #19
  • regenerate o200k encoding from tiktoken file (c7ba009)

2.2.0

17 Jul 23:04
5c51087
Compare
Choose a tag to compare

2.2.0 (2024-07-17)

Bug Fixes

  • add correct encoding for o200k_base (137e07b)
  • add gpt-4o on readme as supported model (27b4e20)
  • update readme (0b33e1e)

Features

  • add o200k_base test plans (44ce38e)
  • added o200k_base to encodings and configured it's specialTokens (2a9da2b)

2.1.2

07 Oct 00:57
Compare
Choose a tag to compare

2.1.2 (2023-10-07)

Bug Fixes

  • bind encodeChat and encodeChatGenerator (86c270c), closes #15

2.1.1

01 Jun 08:05
Compare
Choose a tag to compare

2.1.1 (2023-06-01)

Bug Fixes

  • missing default exports due to limitations of ESM (2a55474), closes #11

2.1.0

01 Jun 07:42
Compare
Choose a tag to compare

2.1.0 (2023-06-01)

Bug Fixes

  • correct excluded path (71af9d3)
  • workaround for webpack not exposing the default export in UMD correctly (84887b4), closes #12

Features

2.0.0

24 May 02:03
774cf36
Compare
Choose a tag to compare

2.0.0 (2023-05-24)

Features

  • complete rewrite to support different models (eedd944), closes #5 #6
  • more tests and better README (e660a25)

BREAKING CHANGES

  • default encoder is now GPT3.5 / GPT4

2.0.0-beta.2

24 May 01:40
Compare
Choose a tag to compare
2.0.0-beta.2 Pre-release
Pre-release

2.0.0-beta.2 (2023-05-24)

Features

  • more tests and better README (e660a25)

2.0.0-beta.1

23 May 16:59
Compare
Choose a tag to compare
2.0.0-beta.1 Pre-release
Pre-release

2.0.0-beta.1 (2023-05-23)

Features

  • complete rewrite to support different models (eedd944), closes #5 #6

BREAKING CHANGES

  • default encoder is now GPT3.5 / GPT4

1.0.5

16 May 07:19
f7df7e9
Compare
Choose a tag to compare

1.0.5 (2023-05-16)

Bug Fixes

1.0.4

01 May 05:44
Compare
Choose a tag to compare

1.0.4 (2023-05-01)

Bug Fixes

  • eslint and jest complaints (8b1cb01)
  • importing/requiring under certain setups was not working (8b71131), closes #4