Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid loading entire dataset by getting the nbytes in an array #7356

Merged
merged 7 commits into from
Dec 12, 2022
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions doc/whats-new.rst
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,9 @@ Deprecations
Bug fixes
~~~~~~~~~

- Accessing the property ``.nbytes`` of a DataArray, or Variable no longer
accidentally triggers loading the variable into memory.


Documentation
~~~~~~~~~~~~~
Expand Down
4 changes: 2 additions & 2 deletions xarray/core/variable.py
Original file line number Diff line number Diff line change
Expand Up @@ -403,8 +403,8 @@ def nbytes(self) -> int:
If the underlying data array does not include ``nbytes``, estimates
the bytes consumed based on the ``size`` and ``dtype``.
"""
if hasattr(self.data, "nbytes"):
return self.data.nbytes
if hasattr(self._data, "nbytes"):
dcherian marked this conversation as resolved.
Show resolved Hide resolved
return self._data.nbytes
else:
return self.size * self.dtype.itemsize

Expand Down
16 changes: 16 additions & 0 deletions xarray/tests/test_dataarray.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@
from xarray.core.types import QueryEngineOptions, QueryParserOptions
from xarray.core.utils import is_scalar
from xarray.tests import (
InaccessibleArray,
ReturnItem,
assert_allclose,
assert_array_equal,
Expand Down Expand Up @@ -3277,6 +3278,21 @@ def test_from_multiindex_series_sparse(self) -> None:

np.testing.assert_equal(actual_coords, expected_coords)

def test_nbytes_does_not_load_data(self) -> None:
array = InaccessibleArray(np.zeros((3, 3), dtype="uint8"))
da = xr.DataArray(array, dims=["x", "y"])

# If xarray tries to instantiate the InaccessibleArray to compute
# nbytes, the following will raise an error.
# However, it should still be able to accurately give us information
# about the number of bytes from the metadata
assert da.nbytes == 9
# Here we confirm that this does not depend on array having the
# nbytes property, since it isn't really required by the array
# interface. nbytes is more a property of arrays that have been
# cast to numpy arrays.
assert not hasattr(array, "nbytes")

def test_to_and_from_empty_series(self) -> None:
# GH697
expected = pd.Series([], dtype=np.float64)
Expand Down