-
Notifications
You must be signed in to change notification settings - Fork 29
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Dynamic Arrayset Backend Update and Manual Configuration of Filter Options #133
Conversation
Codecov Report
@@ Coverage Diff @@
## master #133 +/- ##
==========================================
+ Coverage 92.81% 93.26% +0.45%
==========================================
Files 61 62 +1
Lines 10608 10923 +315
Branches 1041 1059 +18
==========================================
+ Hits 9845 10187 +342
+ Misses 555 518 -37
- Partials 208 218 +10
|
@elistevens, care to take a look at this and let me know what you think? I've got a local branch with tests, but before finalizing that I want to make sure the API and utility is what we would expect. |
The API is functional, but doesn't strike me as friendly. A couple things that would make it more user-friendly:
backend = '00'
backend = 'hdf5'
backend = {
'name': 'hdf5',
'complib': 'blosc:lz4',
'complevel': 3,
'fletcher32': True,
}
co.arraysets.init_arrayset(
'train_images',
prototype=sample_trimg,
backend=backend)
dset_trimgs.change_default_backend(backend)
dset_trimgs.change_default_backend(**backend) I'm not usually a fan of arguments that can be different types, but I think it makes sense here. |
Ok. this makes sense. Will look at it from the ground up before attempting to merge. |
eed99ab
to
ef254d6
Compare
… calls which allow backend and opts to by changed at any time in the future, without rewriting all the data contained in an arrayset
ef254d6
to
990536a
Compare
So I've gone another round at this.
Done! The format i've gone with is: # for backend specification, but default options
co.arraysets.init_arrayset('foo', prototype=np.arange(10), backend_opts='00')
# for backend specification, and manual options
opts = {
'backend': '00',
'shuffle': 'byte',
'complib': 'blosc:zstd',
'complevel': 5,
'fletcher32': True,
}
co.arraysets.init_arrayset('foo', prototype=np.arange(10), backend_opts=opts)
# required opts depend on backend. `numpy` backend requires none, so
co.arraysets.init_arrayset('foo', prototype=np.arange(10), backend_opts='10')
# is equivalent to
co.arraysets.init_arrayset('foo', prototype=np.arange(10), backend_opts={'backend': '10'})
I'd like to hold off on this one for now because:
I think it's a nice quality of life addition, but we need a real configuration management solution on the client before going down this road. How much of a pain will this be for you? (I'm probably being quite myopic here...)
TBD. The I know it's a trivial mapping to make, but it's not necessary right now, and can be added in the future when there's more then 2-3 backends |
Motivation and Context
Why is this change required? What problem does it solve?:
Description
Describe your changes in detail:
Screenshots (if appropriate):
Types of changes
What types of changes does your code introduce? Put an
x
in all the boxes that apply:Is this PR ready for review, or a work in progress?
How Has This Been Tested?
Put an
x
in the boxes that apply:Checklist: