IterDictIndexer can index pre-tokenised documents

[cmacdonald]

This PR adds support for pretokenised indices for Terrier, and is now ready for review.

To run the tests, the following command can be used:
To use this, the pretoks branch of Terrier must be installed:

git clone https://github.com/terrier-org/terrier-core.git
cd terrier-core
git checkout --track origin/pretoks
mvn -DskipTests install

Further, the terrier-python-helper in pyterrier must be updated:

cd pyterrier/terrier-python-helper
mvn install

To run the tests, the following invocation is used
TERRIER_VERSION=5.7-SNAPSHOT TERRIER_HELPER_VERSION=0.0.7 pytest -s tests/test_iterdictindex_pretok.py

Usage looks like:

indexer = pt.IterDictIndexer("./index", toks=True)
indexer.index( [
  {'docno' : 'd1', 'toks' : {'a' : 1, 'b' : 2}
])

NB: There is no fields support for this indexer.

NBB: The Github Actions will not pass without the upstream changes in Terrier being merged. The corresponding PR for the Terrier changes is terrier-org/terrier-core#204

[cmacdonald]

We also need to ensure that /all/ test pass, and this codebase continues to work with Terrier 5.6.

[seanmacavaney]

Looks good to me.

I fixed a syntax error, removed an unused argument, and added code that forces all values to ints. It's worth taking a peek at those changes.

[cmacdonald]

Good spots. Are we happy with pretokenised=True as being the appropriate kwarg? I guess some documentation also needed, including examples e.g. BERTTokenizer

[seanmacavaney]

I think I’d still rather just have it detect whether the indexed fields are dicts and act accordingly. But IIRC this proposal was previously rejected.

I’m not sure what argument name would be most memorable. pretokenised seems alright, but could trip folks up with z vs s. I guess toks solved that, but is far less descriptive.

Yeah, having some example tokenisers is a good idea. Perhaps an example using Counter to count the WP’s too?

[cmacdonald]

All tests pass, but pretokenised tests are skipped until the new helper has been released to Maven.