Skip to content

Tokenizer 1.26.1
Compare
Choose a tag to compare
@guillaumekln guillaumekln released this 31 May 10:54
· 92 commits to master since this release
v1.26.1
48d82ab

Fixes and improvements

  • Fix application of the BPE vocabulary when using preserve_segmented_tokens and a subword appears without joiner in the vocabulary
  • Fix compilation with ICU versions older than 60
Assets 2
Loading