Fixed a bug that in the reader pipline, the document count is always less that the actual number of documents by the number of files. #286
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I came across this bug where I was running comparison of datasets of same number of documents but split into different shards. After inspection of the code, I found that in the code below the number got sent into the documents stat is actually the index of the last docuemnt in a given file, which is less than the number of documents in the file by 1.
datatrove/src/datatrove/pipeline/readers/base.py
Lines 193 to 204 in c2fc902
Therefore, I'm proposing a simple fix that just increment the variable
di
when it's non-zero.