Update LitData version and restore previous LitData assertions in tests #1609
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rasbt
approved these changes
Jul 22, 2024
| chunk_bytes="200MB", | ||
| ) | ||
| optimize( | ||
| fn=partial(tokenize, split_dataset["val"]), | ||
| inputs=list(range(len(split_dataset["val"]))), | ||
| output_dir=self.data_path_val, | ||
| num_workers=(os.cpu_count() - 1), | ||
| num_workers=min(8, os.cpu_count() - 1), |
There was a problem hiding this comment.
That's a good change. There were a couple of people who reported issues where everything hangs / gets very slow, and it might have been due to too many workers and processes.
There was a problem hiding this comment.
Sure, but it's not why I did it. It's just that in this dataset, there aren't that many documents. So it's wasteful to create more workers than there are items to process, some workers would just not get anything to do. Like on the A100 we have 256 cores but that's too much for this dataset.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Restores the assertions to what they were before updating LitData in #1573. LitData 0.2.17 has fixes for handling drop_last, which was causing the previous changes in the assertions and number of samples returned.
This also addresses the comment from @carmocca #1573 (comment)