NetCDF Error. NetCDF: Not a valid data type or _FillValue mismatch

[croachutas]

I've recently installed CMS on a 'new' workstation (actually second hand hardware but a fresh install of Ubuntu 18.04.3). I compiled the various dependencies (HDF5, szip, zlib, netcdf C and netcdf fortran) from their source codes following instructions on the HDF5 and NetCDF websites before building CMS.

The test case works when set to ASCII output but switching it to netcdf output (mostly since I'd prefer less post-processing when running several million particles in my actual experiment) I get an error:
Error in the netCDF file: expt_example/SCRATCH/traj_file_1.nc
NetCDF: Not a valid data type or _FillValue type mismatch.

The same error occurs when running CMS with velocity fields converted to CMS format locally rather than downloaded remotely.

There are no issues when converting netCDF files to the CMS setup using getData, so this error seems to be exclusively related to how CMS writes NetCDF files out.

I'm currently using HDF5 v1.10.5, NetCDF C v4.7.2 and NetCDF Fortran v4.5.2

Doing a little googling the same error output has been reported with recent versions of netcdf-4python (Unidata/netcdf4-python#929), and is apparently related to NetCDF-C 4.6.2 and onward being stricter than earlier versions about FillValue and Variable types matching.

I suspect it'd be possible to downgrade to an older version of NetCDF as a short term work around (or put up with the extra post-processing and HD-space demands from going with ASCII). However, for future compatibility you probably should double check the types you're assigning to the FIllValue...

[croachutas]

Can confirm downgrading to NetCDF C 4.6.1 and NetCDF Fortran 4.4.5 fixes this issue on my machine. Still, probably do wanna fix it properly in the CMS code to avoid it reoccurring for other people.

[beatrixparis]

Thank you Christopher, this is very helpful; we will look into a fix. Claire

…

Sent from my iPhone Claire Beatrix Paris-Limouzy, PhD Rosenstiel School of Marine and Atmospheric Science

________________________________ De : Christopher Roach <notifications@github.com> Envoyé : Tuesday, December 3, 2019 5:29:31 AM À : beatrixparis/connectivity-modeling-system <connectivity-modeling-system@noreply.github.com> Cc : Subscribed <subscribed@noreply.github.com> Objet : Re: [beatrixparis/connectivity-modeling-system] NetCDF Error. NetCDF: Not a valid data type or _FillValue mismatch (#38) Can confirm downgrading to NetCDF C 4.6.1 and NetCDF Fortran 4.4.5 fixes this issue on my machine. Still, probably do wanna fix it properly in the CMS code to avoid it reoccurring for other people. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fbeatrixparis%2Fconnectivity-modeling-system%2Fissues%2F38%3Femail_source%3Dnotifications%26email_token%3DACZ27ZHJVJXAKYMGUE47XVTQWYYIXA5CNFSM4JN5FJO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFY4ECQ%23issuecomment-561103370&data=02%7C01%7Ccparis%40rsmas.miami.edu%7C0521106846aa4cdbd34408d777dbaf67%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C637109657739228613&sdata=uinsr6sZ6dGZ8YGXeefiyHg21S4RTHkzYs0%2FIpFDWSk%3D&reserved=0>, or unsubscribe<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FACZ27ZEHQODSE2ERPB433ZLQWYYIXANCNFSM4JN5FJOQ&data=02%7C01%7Ccparis%40rsmas.miami.edu%7C0521106846aa4cdbd34408d777dbaf67%7C2a144b72f23942d48c0e6f0f17c48e33%7C0%7C0%7C637109657739233594&sdata=1IeeMMZfZvqU9dsTBLgM9fMbF8vBVicJbT12XlpxHfU%3D&reserved=0>.

[benfarmer05]

I am still receiving this error today, using an HPC with netcdf-c/4.7.3 and netcdf-fortran/4.5.2 installed. Outputting to ASCII solved the error.