-
Notifications
You must be signed in to change notification settings - Fork 138
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve local temp directory error when only remote
is specified
#683
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Can we please add a unit test for the same? :D |
added unit test @karan6181 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can merge the PR after the CI passes.
Co-authored-by: Karan Jariwala <karankjariwala@gmail.com>
@karan6181 @XiaohanZhangCMU I was trying to fix the test that I wrote, but I realized there's also a test_existing_local_raises_exception in test_stream.py that tests this already. Are we good with merging the PR without me adding an additional test then? |
Yes, it's okay to merge since the unit test is already in place. To make CI pass, you might want to change the exception type from |
Description of changes:
Improves the local temp directory error when only a
remote
path is specified. We want to make sure that iflocal
is unspecified, we still create an entirely new temporary directory to hold the shard files, but if a user wants to reuse their locally cached dataset, they should explicitly specify thelocal
argument.Issue #, if available:
Merge Checklist:
Put an
x
without space in the boxes that apply. If you are unsure about any checklist, please don't hesitate to ask. We are here to help! This is simply a reminder of what we are going to look for before merging your pull request.General
Tests
pre-commit
on my change. (check out thepre-commit
section of prerequisites)