Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dataset.map #3459

Merged
merged 20 commits into from
Nov 9, 2019
Merged

Dataset.map #3459

merged 20 commits into from
Nov 9, 2019

Conversation

max-sixty
Copy link
Collaborator

@max-sixty max-sixty commented Oct 29, 2019

This is the first step towards organizing these functions a bit better, as outlined in #1251, #1618, #2674

While this one is a fairly easy decision, others are going to be harder. Open to a more methodical up-front process before we start making changes, if we think that's necessary.

@max-sixty
Copy link
Collaborator Author

I'm getting bemuisng test failures here! All the tests apart from docs say they pass, Docs says Cancelled after 60m. There are no explicit changes to docs here, and I can't imagine any implicit changes. If anyone has any bright ideas, help is welcome!

@max-sixty
Copy link
Collaborator Author

I see this in the GH Checks page (but, again, no test runs failed, apparently)

self = <xarray.tests.test_backends.TestPydapOnline object at 0x7fd9047f8518>

    def test_cmp_local_file(self):
>       with self.create_datasets() as (actual, expected):

xarray/tests/test_backends.py:3137: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/contextlib.py:81: in __enter__
    return next(self.gen)
xarray/tests/test_backends.py:3191: in create_datasets
    actual = open_dataset(url, engine="pydap", **kwargs)
xarray/backends/api.py:504: in open_dataset
    store = backends.PydapDataStore.open(filename_or_obj, **backend_kwargs)
xarray/backends/pydap_.py:78: in open
    ds = pydap.client.open_url(url, session=session)
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/pydap/client.py:67: in open_url
    timeout).dataset
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/pydap/handlers/dap.py:54: in __init__
    raise_for_status(r)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

response = <Response at 0x7fd9045ac588 500 Internal Server Error>

    def raise_for_status(response):
        # Raise error if status is above 300:
        if response.status_code >= 300:
            raise HTTPError(
                detail=response.status+'\n'+response.text,
                headers=response.headers,
>               comment=response.body
            )
E           webob.exc.HTTPError: 500 Internal Server Error
E           Error { 
E               code = 500;
E               message = "Problem encountered with BES connection. Message: 'Failed to write to socket: Broken pipe (Write failed)' OPeNDAPClient executed 131 prior commands.";
E           }

/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/pydap/net.py:39: HTTPError

@max-sixty
Copy link
Collaborator Author

I think docs stopped building and just froze (so timed out rather than erroring), and so the error wasn't reported up to Azure Pipelines. I got this from the log:

>>>-------------------------------------------------------------------------
Warning in /home/vsts/work/1/s/doc/computation.rst at block ending on line 472
Specify :okwarning: as an option in the ipython:: block to suppress this message
----------------------------------------------------------------------------
/usr/share/miniconda/envs/xarray-tests/bin/sphinx-build:1: DeprecationWarning: Dataset.apply is deprecated in favor of Dataset.map and will be changed or removed in a future version of xarray
  #!/usr/share/miniconda/envs/xarray-tests/bin/python3.7
<<<-------------------------------------------------------------------------
/home/vsts/work/1/s/xarray/core/dataarray.py:docstring of xarray.DataArray.integrate:12: WARNING: Unexpected indentation.
/home/vsts/work/1/s/xarray/core/dataarray.py:docstring of xarray.DataArray.interp:20: WARNING: Inline strong start-string without end-string.
/home/vsts/work/1/s/xarray/core/dataarray.py:docstring of xarray.DataArray.interpolate_na:8: WARNING: Definition list ends without a blank line; unexpected unindent.

Sphinx parallel build error:
RuntimeError: Non Expected warning in `/home/vsts/work/1/s/doc/computation.rst` line 472

@max-sixty
Copy link
Collaborator Author

max-sixty commented Nov 1, 2019

This is ready for review, I'll add a whatsnew depending on whether we decide to do the same for GroupBy

@max-sixty
Copy link
Collaborator Author

Ready to merge!

Copy link
Member

@shoyer shoyer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looks good to me, but let's keep around docstrings for map methods.

keep_attrs: bool = None,
args: Iterable[Any] = (),
**kwargs: Any,
) -> "Dataset":
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we keep the docstring here, even if it's just "Alias for Dataset.map"?

Ideally we would use the See also section, which gets turned into a link by numpydoc.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also added See Also for the drop cases

@max-sixty
Copy link
Collaborator Author

Lmk if that was an "OK to merge" @shoyer ! Thanks

@shoyer
Copy link
Member

shoyer commented Nov 9, 2019

Looks good to me!

@max-sixty max-sixty merged commit db0f13d into pydata:master Nov 9, 2019
@max-sixty max-sixty deleted the dataset-map branch November 9, 2019 21:10
dcherian added a commit to dcherian/xarray that referenced this pull request Nov 12, 2019
* upstream/master:
  add missing pint integration tests (pydata#3508)
  DOC: update bottleneck repo url (pydata#3507)
  add drop_sel, drop_vars, map to api.rst (pydata#3506)
  remove syntax warning (pydata#3505)
  Dataset.map, GroupBy.map, Resample.map (pydata#3459)
  tests for datasets with units (pydata#3447)
  fix pandas-dev tests (pydata#3491)
  unpin pseudonetcdf (pydata#3496)
  whatsnew corrections (pydata#3494)
dcherian added a commit to dcherian/xarray that referenced this pull request Nov 12, 2019
* upstream/master:
  add missing pint integration tests (pydata#3508)
  DOC: update bottleneck repo url (pydata#3507)
  add drop_sel, drop_vars, map to api.rst (pydata#3506)
  remove syntax warning (pydata#3505)
  Dataset.map, GroupBy.map, Resample.map (pydata#3459)
  tests for datasets with units (pydata#3447)
  fix pandas-dev tests (pydata#3491)
  unpin pseudonetcdf (pydata#3496)
  whatsnew corrections (pydata#3494)
  drop_vars; deprecate drop for variables (pydata#3475)
  uamiv test using only raw uamiv variables (pydata#3485)
  Optimize dask array equality checks. (pydata#3453)
dcherian added a commit to dcherian/xarray that referenced this pull request Nov 13, 2019
* upstream/master:
  format indexing.rst code with black (pydata#3511)
  add missing pint integration tests (pydata#3508)
  DOC: update bottleneck repo url (pydata#3507)
  add drop_sel, drop_vars, map to api.rst (pydata#3506)
  remove syntax warning (pydata#3505)
  Dataset.map, GroupBy.map, Resample.map (pydata#3459)
  tests for datasets with units (pydata#3447)
  fix pandas-dev tests (pydata#3491)
  unpin pseudonetcdf (pydata#3496)
  whatsnew corrections (pydata#3494)
  drop_vars; deprecate drop for variables (pydata#3475)
  uamiv test using only raw uamiv variables (pydata#3485)
  Optimize dask array equality checks. (pydata#3453)
  Propagate indexes in DataArray binary operations. (pydata#3481)
  python 3.8 tests (pydata#3477)
dcherian added a commit to dcherian/xarray that referenced this pull request Nov 13, 2019
commit d430ae0
Author: dcherian <deepak@cherian.net>
Date:   Wed Nov 13 08:27:04 2019 -0700

    proper fix.

commit 7fd69be
Author: dcherian <deepak@cherian.net>
Date:   Wed Nov 13 08:03:26 2019 -0700

    fix whats-new merge.

commit 4489394
Merge: 279ff1d b74f80c
Author: dcherian <deepak@cherian.net>
Date:   Wed Nov 13 08:03:06 2019 -0700

    Merge remote-tracking branch 'upstream/master' into fix/plot-broadcast

    * upstream/master:
      format indexing.rst code with black (pydata#3511)
      add missing pint integration tests (pydata#3508)
      DOC: update bottleneck repo url (pydata#3507)
      add drop_sel, drop_vars, map to api.rst (pydata#3506)
      remove syntax warning (pydata#3505)
      Dataset.map, GroupBy.map, Resample.map (pydata#3459)
      tests for datasets with units (pydata#3447)
      fix pandas-dev tests (pydata#3491)
      unpin pseudonetcdf (pydata#3496)
      whatsnew corrections (pydata#3494)
      drop_vars; deprecate drop for variables (pydata#3475)
      uamiv test using only raw uamiv variables (pydata#3485)
      Optimize dask array equality checks. (pydata#3453)
      Propagate indexes in DataArray binary operations. (pydata#3481)
      python 3.8 tests (pydata#3477)

commit 279ff1d
Author: dcherian <deepak@cherian.net>
Date:   Wed Nov 13 08:02:44 2019 -0700

    Undo the transpose change and add test to make sure transposition is right.

commit c9cc698
Author: dcherian <deepak@cherian.net>
Date:   Wed Nov 13 08:01:39 2019 -0700

    Test to make sure transpose is right

commit 9b35ecf
Author: dcherian <deepak@cherian.net>
Date:   Sat Nov 2 15:49:08 2019 -0600

    Additional test.

commit 7aed950
Author: dcherian <deepak@cherian.net>
Date:   Sat Nov 2 15:20:07 2019 -0600

    make plotting work with transposed nondim coords.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants