-
Notifications
You must be signed in to change notification settings - Fork 541
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[0.1.0] Intro NDArryLoader #75
Conversation
Apply Sweep Rules to your PR?
|
@MooooCat |
Sure. |
Will update user guide before official release |
I have a question about the NDArray Loader. I noticed that it names the .npy in order. (I'm not sure if this will happen yet) When multiple data loaders run at the same time using default parameters (that is, the same cache_dir), this sequential naming method may cause conflicts: NDArrayLoader 1 stores In the naming of cached .npy files, if the path contains an id field, there will be no such concern. Also, suggesting developers to use different cache paths is obviously a right approach. Could this be a problem? |
@MooooCat Yes, different NDArrayLoader should use different subdirs. I'll draft a PR fot it |
Description
Motivation and Context
NDArrayLoader
for lazy reading of columnarly stored ndarryHow has this been tested?
Types of changes
Checklist: