added corpus_iter for Terrier index #426
Conversation
pyterrier/bootstrap.py
Outdated
|
|
||
| for skipped in range(0, direct_inputstream.getEntriesSkipped()): | ||
| meta = meta_inputstream.next() | ||
| yield {k : meta[keys_offset[k]] for k in keys_offset} |
There was a problem hiding this comment.
Not super familar with the semantics of getEntriesSkipped. But should this include an empty toks to ensure that every dict from the iter has the same fields?
There was a problem hiding this comment.
Not super familar with the semantics of getEntriesSkipped.
direct_inputstream only address documents that are non-empty. So we have to address non-empty documents using getEntriesSkipped()
pyterrier/bootstrap.py
Outdated
| @@ -271,14 +271,55 @@ def _index_add(self, other): | |||
| raise ValueError("Cannot document-wise merge indices with and without positions (%r vs %r)" % (blocks_1, blocks_2)) | |||
| multiindex_cls = autoclass("org.terrier.realtime.multi.MultiIndex") | |||
| return multiindex_cls([self, other], blocks_1, fields_1 > 0) | |||
|
|
|||
| def _index_corpusiter(self, direct=True): | |||
There was a problem hiding this comment.
direct isn't the most intuitive name to me for this field. Perhaps return_toks?
There was a problem hiding this comment.
or toks? The corresponding indexing arg is pretokenised?
Final decision: |
|
I don't think I feel super strongly one way or the other, but I feel |
|
Ok, ta. I write some unit tests then I merge. |
|
My revised implementation requires Python 3.8. Colab is Py 3.10 now |
|
I'm happy with min python version of 3.8 |
Initial draft of #425
Unit tests are needed.
@seanmacavaney is the API useful?