-
-
Notifications
You must be signed in to change notification settings - Fork 278
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
open_array()
: adding the meta_array
argument
#1396
Conversation
Codecov Report
@@ Coverage Diff @@
## main #1396 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 37 37
Lines 14857 14866 +9
=========================================
+ Hits 14857 14866 +9
|
@rabernat i think this is the last piece needed for xarray/GDS support |
Some quick numbers (all IO reads): | | Compression Ratio | Throughput | ----------- | :-----------: |-----------: | | POSIX | n/a | 1.63 GiB/s | | KvikIO (compat mode) | n/a | 8.43 GiB/s | | Zarr (no compression) | n/a |8.00 GiB/s | | Zarr+Snappy | 1000/322 | 10.57 GiB/s| | Zarr+Gdeflate | 1000/27 | 29.05 GiB/s| | Zarr+LZ4 | 1000/9 | 48.95 GiB/s| Notice, this is just a quick benchmark to show the potential of nvCOMP. When Zarr v2.15 is released and #131 merged, we can start optimizing. Additionally, when zarr-developers/zarr-python#1396 is merged, we can introduce a wrapper for `zarr.open()` that seamlessly reads/writes to/from GPU memory and do on-the-fly compression. Authors: - Mads R. B. Kristensen (https://github.com/madsbk) Approvers: - Benjamin Zaitlen (https://github.com/quasiben) URL: #208
Would it be possible to include this in the 2.15 Zarr release ? |
Makes sense to me to have it with your other PR, @madsbk. (Sidenote: 2.15a1 just went out) |
Agree, it would be great to get this in |
@madsbk: quick higher-level question -- how many other locations do you think might need this? |
I am not sure, but with this PR merged all functions in |
Anything that needs to be done here? |
Hi @madsbk. Apologies. I took a look after your last push, but the Windows 3.11 build was failing and a re-run didn't fix it (and then I promptly got side-tracked). I just re-ran and all is green. Let's get this in. I'll push out an immediate pre-release and then review the state of everything tomorrow. (Feedback on a full release of 2.15.0 welcome from any/all.) Thanks for hanging in there! |
Awesome, thanks @joshmoore ! |
Like the other creation functions,
open_array()
now takes themeta_array
argument.TODO: