Granular BitGroom feature for netcdf-c

[czender]

Granular BitGroom (GBG) combines features of BitGroom, BitRound by Kouznetsov (2020), and DigitRound by Delaunay et al. (2019). GBG improves compression ratios by ~20% relative to BitGroom for NSD=3 on our benchmark 1 GB climate model output dataset. Its invocation is identical to BitGroom, so this patchset mainly utilizes a new enumerated value of the quantize_mode flag to invoke the new algorithm. No tests (yet) in this patchset. For correctness, GBG can be compared to current implementations in NCO and CCR.

[edwardhartnett]

@czender do you have cites for the two papers your reference?

[czender]

@edwardhartnett If you mean the following two, then yes, and they are also on the CCR homepage:

Delaunay, X., A. Courtois, and F. Gouillon (2019), Evaluation of
lossless and lossy algorithms for the compression of scientific
datasets in netCDF-4 or HDF5 files, Geosci. Model Dev., 12(9),
4099-4113, doi:10.5194/gmd-2018-250

Kouznetsov, R. (2021), A note on precision-preserving compression of scientific data, Geosci. Model Dev., 14(1), 377-389, https://doi.org/10.5194/gmd-14-377-2021

[DennisHeimbigner]

Is the number of algorithms likely to grow significantly,
If so then I must object to having a different attribute name for every algorithm.

[edwardhartnett]

I believe not more than a few algorithms are envisioned, right @czender ?

But I wonder why we don't just take the best algorithm and use that? Do we need the old one, once an improvement has come along?

[czender]

@DennisHeimbigner I do not intend to try to add anymore quantization algorithms. After BitGroom was published in 2016, it inspired others to optimize quantization algorithms even further, within the constraints of guaranteeing the user-specified NSD and keeping IEEE on-disk format so no decoder necessary. This GBG algorithm incorporates that progress, which significantly improves compression ratio (CR), something like 20% better than BG for NSD=3 (followed by DEFLATE). The "low-hanging fruit" are in GBG, so I think algorithmic improvements cannot improve GBG CR by more than 5% without violating the above constraints. I thought it important that netcdf-c not be limited to BG given the known improvements that were possible, so once Ed put in BG, it prompted me to develop and submit a "best of" algorithm to netcdf-c rather than see it languish in NCO or CCR.
@edwardhartnett Offering multiple quantization options to users in netcdf-c may not be helpful to anyone. Most users will not understand the trade-offs of different algorithms, and those who really care or wish to intercompare algorithms could use NCO or CCR. Nevertheless, "we" decided that a quantize_mode enum was wise, so the natural path for GBG was to use that. I understand Dennis' reservation about attribute name proliferation. The options seem to be to 1) Use the existing attribute name convention but limit the number of algorithms. 2) Change the convention so that all NSD-based quantization algorithms use the same attribute name. 3) A combination of 1 and 2 where netcdf-c only has one quantization algorithm (and thus one attribute name), and allow the algorithm to change incrementally under the hood when justified. Any of those seem fine to me. FWIW, NCO quantization defaults to GBG now.
GBG yields better CR than BG for small NSD so its accuracy is corresponding worse than BG. However, users essentially pick the desired quantization error when they choose NSD for a variable. Is there any point to giving users two choices (algorithm and NSD) instead of one (NSD)? Maybe, maybe not. Anyway, this is clearly an important topic to seek consensus on before releasing 4.8.2. Thanks for raising these questions.

[WardF]

Clearing out the PR backlog, it appears that the quantize test is giving an 'unexpected error'. The autotools-based test is silent, I will adjust that to provide additional information in the case of failure, but the cmake test is at least a little bit more verbose about it.

[czender]

Thanks, @WardF. I see why the test fails and it's an easy fix. It fails because it expects an error if NC_QUANTIZE_BITGROOM is NOT the last quantize mode defined. Now NC_QUANTIZE_GRANULARBG is the last quantize mode defined, so the test that previously passed should now fail, as observed. Not sure why this was silently passing in the autoconf-based testing I did earlier, though. I think the simplest fix is to change the test so it tries to access an undefined quantize mode one greater than NC_QUANTIZE_GRANULARBG, and I will submit a patch for that soon. Another route would be to add another token, e.g., NC_QUANTIZE_MODE_MAX defined as the greatest valid enumerated value of QUANTIZE modes. I hesitate to do that because my sense is that adding tokens is frowned upon.