-
Notifications
You must be signed in to change notification settings - Fork 0
/
DESCRIPTION
20 lines (20 loc) · 1.16 KB
/
DESCRIPTION
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Package: tokenizers.bpe
Type: Package
Title: Byte Pair Encoding Text Tokenization
Version: 0.1.3
Authors@R: c(person('Jan', 'Wijffels', role = c('aut', 'cre', 'cph'), email = 'jwijffels@bnosac.be', comment = "R wrapper"),
person('BNOSAC', role = 'cph', comment = "R wrapper"),
person('VK.com', role = 'cph'),
person('Gregory Popovitch', role = c('ctb', 'cph'), comment = "Files at src/parallel_hashmap (Apache License, Version 2.0"),
person('The Abseil Authors', role = c('ctb', 'cph'), comment = "Files at src/parallel_hashmap (Apache License, Version 2.0"),
person('Ivan Belonogov', role = c('ctb', 'cph'), email = 'xbelonogov@gmail.com', comment = "Files at src/youtokentome (MIT License)"))
Maintainer: Jan Wijffels <jwijffels@bnosac.be>
Description: Unsupervised text tokenizer focused on computational efficiency. Wraps the 'YouTokenToMe' library <https://github.com/VKCOM/YouTokenToMe> which is an implementation of fast Byte Pair Encoding (BPE) <https://aclanthology.org/P16-1162/>.
URL: https://github.com/bnosac/tokenizers.bpe
License: MPL-2.0
Encoding: UTF-8
LazyData: true
RoxygenNote: 7.1.2
Depends: R (>= 2.10)
Imports: Rcpp (>= 0.11.5)
LinkingTo: Rcpp