Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dimension Co-ordinates incorectly saving _FillValue attribute #1865

Closed
ghost opened this issue Jan 29, 2018 · 6 comments · Fixed by #1869
Closed

Dimension Co-ordinates incorectly saving _FillValue attribute #1865

ghost opened this issue Jan 29, 2018 · 6 comments · Fixed by #1869

Comments

@ghost
Copy link

ghost commented Jan 29, 2018

Code Sample, a copy-pastable example if possible

import xarray as xr
import pandas as pd
import numpy as np

temp = 273.15 + 25 * np.random.randn(2, 2)
lon = [0.0, 5.0]
lat = [10.0, 20.0]

ds = xr.Dataset({'temperature': (['lat', 'lon'], temp)},
                coords={'lat': lat,
                        'lon': lon})

ds['lat'].attrs = {
    'standard_name': 'latitude',
    'long_name': 'latitude',
    'units': 'degrees_north',
    'axis': 'Y'}
ds['lon'].attrs = {
    'standard_name': 'longitude',
    'long_name': 'longitude',
    'units': 'degrees_east',
    'axis': 'X'}
ds['temperature'].attrs = {
    'standard_name': 'air_temperature',
    'units': 'K'}
ds.attrs = {
    ('title', 'non-conforming CF 1.6 data produced by xarray 0.10'),
    ('Conventions', 'CF-1.6')}

ds.to_netcdf('/tmp/test.nc')

Problem description

According to the last sentence of the first paragraph of 2.5.1. Missing data, valid and actual range of data in NetCDF Climate and Forecast (CF) Metadata Conventions 1.7:

Missing data is not allowed in coordinate variables.

When I use the conformance checker it issues an INFO message to this point for the co-ordinate variables.

Output of CF-Checker follows...

CHECKING` NetCDF FILE: /tmp/29428.nc
=====================
Using CF Checker Version 3.0.0
Checking against CF Version CF-1.6
Using Standard Name Table Version 48 (2017-11-28T15:32:48Z)
Using Area Type Table Version 6 (22 February 2017)


------------------
Checking variable: temperature
------------------

------------------
Checking variable: lat
------------------
INFO: attribute _FillValue is being used in a non-standard way

------------------
Checking variable: lon
------------------
INFO: attribute _FillValue is being used in a non-standard way

ERRORS detected: 0
WARNINGS given: 0
INFORMATION messages: 2

Expected Output

Co-ordinate variables should not store a _FillValue attribute

Output of xr.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.4.final.0
python-bits: 64
OS: Linux
OS-release: 4.14.15-1-ARCH
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
xarray: 0.10.0
pandas: 0.22.0
numpy: 1.14.0
scipy: 1.0.0
netCDF4: 1.3.1
h5netcdf: 0.5.0
Nio: None
bottleneck: 1.2.1
cyordereddict: None
dask: 0.16.1
matplotlib: 2.1.2
cartopy: None
seaborn: None
setuptools: 38.4.0
pip: 9.0.1
conda: None
pytest: None
IPython: None
sphinx: None

@ghost
Copy link
Author

ghost commented Jan 29, 2018

Please add the label "CF conventions" to this issue

@shoyer
Copy link
Member

shoyer commented Jan 30, 2018

Please take a look at the discussion in #1598.

You can use ds.to_netcdf('/tmp/test.nc', encoding={'lat': {'_FillValue': None}, 'lon': {'_FillValue': None}}) to disable this.

@ghost
Copy link
Author

ghost commented Jan 30, 2018

Thanks.

@ghost ghost closed this as completed Jan 30, 2018
@shoyer
Copy link
Member

shoyer commented Jan 30, 2018

If you have ideas for improving this further (even documentation), please do speak up.

@ghost ghost reopened this Jan 30, 2018
@ghost
Copy link
Author

ghost commented Jan 30, 2018

My apologies for having to re-open the issue, I should have tried the fix before closing.

I get the following error after changing my last line to: ds.to_netcdf('/tmp/test.nc', encoding={'lat': {'_FillValue': None}, 'lon': {'_FillValue': None}}) as suggested.

  File "/path/to/python3.6/site-packages/xarray/backends/netCDF4_.py", line 354, in prepare_variable
    unlimited_dims=unlimited_dims)
  File "/path/to/python3.6/site-packages/xarray/backends/netCDF4_.py", line 176, in _extract_nc4_variable_encoding
    ' %r' % (backend, invalid))
ValueError: unexpected encoding parameters for 'netCDF4' backend:  ['_FillValue']

Do we just need to append '_FillValue' to the list of

     valid_encodings = set(['zlib', 'complevel', 'fletcher32', 'contiguous',
                           'chunksizes', 'shuffle'])
```

@ghost
Copy link
Author

ghost commented Jan 30, 2018

@shoyer, I'll look at the documentation once this is resolved and make a PR.

@ghost ghost mentioned this issue Feb 13, 2018
5 tasks
shoyer pushed a commit that referenced this issue Feb 13, 2018
* Add '_FillValue' to set of valid_encodings for netCDF4 backend

* Add additional omit_fill_value tests

Add additional tests to prevent future regression of setting _FillValue
to None when using the encoding kwarg in to_netcdf.

* Fix additional omit_fill_value tests

Remove copy/paste line that shouldn't have been in a test.
Add additional asserts.
Fix indentation.

* Fix scipy failure in additional omit_fill_value tests

* Added bug-fix documentation for #1865
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant