v0.5.2

BytePairEncoding v0.5.2

Diff since v0.5.1

v0.5.1

BytePairEncoding v0.5.1

Diff since v0.5.0

v0.5.0

BytePairEncoding v0.5.0

Diff since v0.4.0

Merged pull requests:

• support tiktoken (#7) (@chengchingwen)

Closed issues:

• Support for cl100k_base encoding, used by new OpenAI models? (#6)

v0.4.1

BytePairEncoding v0.4.1

Diff since v0.4.0

Merged pull requests:

• support tiktoken (#7) (@chengchingwen)

v0.4.0

BytePairEncoding v0.4.0

Diff since v0.3.2

v0.3.2

BytePairEncoding v0.3.2

Diff since v0.3.1

v0.3.1

BytePairEncoding v0.3.1

Diff since v0.3.0

v0.3.0

BytePairEncoding v0.3.0

Diff since v0.2.0

Merged pull requests:

• adapt textencodebase (#5) (@chengchingwen)

v0.2.0

BytePairEncoding v0.2.0

Diff since v0.1.1

Closed issues:

• Allowing to directly pass dictionary to BPELearner (#1)

Merged pull requests:

• Refactor (#3) (@chengchingwen)

v0.1.1

support unicode normalization