added tokenizer as arg for pt.text.sliding

[mihirs16]

For the purposes of obtaining passages of a given length, tokenisation is performed by using the tokenizer object passed as an argument. The tokenizer object must have a .tokenize(str) -> list[str] method. By default, tokenisation is performed by splitting on one-or-more spaces, i.e. based on the Python regular expression re.compile(r'\s+').

Example

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
pipe = (pt.BatchRetrieve(index, wmodel="DPH", metadata=["docno", "body"])
      >> pt.text.sliding(length=128, stride=64, prepend_attr=None, tokenizer=tok)
      >> pt.text.scorer(wmodel="DPH")
      >> pt.text.max_passage() )

[mihirs16]

yet to discuss the possibility and implementation of sentence segmentation (as part of #217)

[cmacdonald]

Thanks @mihirs16 - we're attending ECIR this week, so reviewing of the PR may be delayed.

[cmacdonald]

Can you add some unit new tests too? You should assert that your requirements on the token lengths are assured.
you may need to add transformers to requirements-test.txt (but ideally, I'd like NOT to add such a heavy dependency), so perhaps you could mock up another tokenizer class.

[mihirs16]

hey! would love further feedback (if any) about this

[cmacdonald]

Hi @mihirs16

Thanks for the ping. Most of the PR looks good. I think that my concern is whether

row[self.text_attr] = ' '.join(toks)

is now appropriate for recovering the passage segment.

Perhaps if the tokenizer is specified, it should call the .decode() method to recover the passage? [NB: Not sure .decode() is the correct method!)

I think also we could have a separate test case that runs IF transformers is installed, and is skipped otherwise:

def test_sliding_tokenize_HGF(self):
  try:
    from transformers import AutoTokenizer
   catch:
    self.skipTest("transformers not installed") 
  ...

[mihirs16]

Sorry for the long post ahead 😅 but most of it is just python shell output.

Thanks for the ping. Most of the PR looks good. I think that my concern is whether

row[self.text_attr] = ' '.join(toks)

is now appropriate for recovering the passage segment.

Perhaps if the tokenizer is specified, it should call the .decode() method to recover the passage? [NB: Not sure .decode() is the correct method!)

@cmacdonald that's a very good point, here's a comparison between the first time a passage is tokenized, and then de-tokenized using ' '.join(toks), and then tokenized again to see if it reproduces the same result... which it does not.

>>> tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
>>> tokenized = tokenizer.tokenize("Perhaps if the tokenizer is specified, it should call the .decode() method to recover the passage?")
>>> tokenized, len(tokenized)
(['perhaps', 'if', 'the', 'token', '##izer', 'is', 'specified', ',', 'it', 'should', 'call', 'the', '.', 'deco', '##de', '(', ')', 'method', 'to', 'recover', 'the', 'passage', '?'], 23)
>>> untokenized = ' '.join(tokenized)
>>> untokenized
'perhaps if the token ##izer is specified , it should call the . deco ##de ( ) method to recover the passage ?'
>>> tokenized_again = tokenizer.tokenize(untokenized)
>>> tokenized_again, len(tokenized_again)
(['perhaps', 'if', 'the', 'token', '#', '#', 'i', '##zer', 'is', 'specified', ',', 'it', 'should', 'call', 'the', '.', 'deco', '#', '#', 'de', '(', ')', 'method', 'to', 'recover', 'the', 'passage', '?'], 28)

As far as i understand, .decode() is useful when we have used .encode() to tokenize and encode the passage. Here's the following example using a huggingface transformer:

>>> encoded = tokenizer.encode("Perhaps if the tokenizer is specified, it should call the .decode() method to recover the passage?")
>>> encoded, len(encoded)
([101, 3383, 2065, 1996, 19204, 17629, 2003, 9675, 1010, 2009, 2323, 2655, 1996, 1012, 21933, 3207, 1006, 1007, 4118, 2000, 8980, 1996, 6019, 1029, 102], 25)
>>> decoded = tokenizer.decode(encoded)
>>> decoded
'[CLS] perhaps if the tokenizer is specified, it should call the. decode ( ) method to recover the passage? [SEP]'

Otherwise, if we use the .tokenize() method, there are other utilities such as .convert_tokens_to_string() to get the original input again. Here's an example:

>>> tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
>>> tokenized = tokenizer.tokenize("Perhaps if the tokenizer is specified, it should call the .decode() method to recover the passage?")
>>> tokenized, len(tokenized)
(['perhaps', 'if', 'the', 'token', '##izer', 'is', 'specified', ',', 'it', 'should', 'call', 'the', '.', 'deco', '##de', '(', ')', 'method', 'to', 'recover', 'the', 'passage', '?'], 23)
>>> untokenized = tokenizer.convert_tokens_to_string(tokenized)
>>> untokenized
'perhaps if the tokenizer is specified, it should call the. decode ( ) method to recover the passage?'
>>> tokenized_again = tokenizer.tokenize(untokenized)
>>> tokenized_again, len(tokenized_again)
(['perhaps', 'if', 'the', 'token', '##izer', 'is', 'specified', ',', 'it', 'should', 'call', 'the', '.', 'deco', '##de', '(', ')', 'method', 'to', 'recover', 'the', 'passage', '?'], 23)

In the light of this, I suppose the Passager class should have a detokenize to handle this in the same way as tokenize?

(using .convert_tokens_to_string() and making it a mandatory attribute, just like tokenize() for the tokenizer passed to SlidingWindowPassager).

[cmacdonald]

I think the type of members of toks could be string or int.

So I think we can roughly do:

toks = tok.tokenize(text)
reconstructed = tok.convert_tokens_to_string(toks)

[mihirs16]

i took the liberty of setting tokenize and detokenize functions in the constructor itself, and also added a separate unittest to handle the case where transformers are installed (but haven't added it to the requirements-test.txt)

[mihirs16]

@cmacdonald would love some feedback on this!

[cmacdonald]

Sorry for the delay on this. I have left a few comments, but its nearly there :-)

[cmacdonald]

Thanks for this @mihirs16!