Maintenence: update mybinder environment.yml dependency versions

[GenevieveBuckley]

I noticed that the mybinder environment.yml file pins dask to version 0.20, but the latest dask release is now up to 1.1.2. It's probably time to update or unpin some of these dependencies. Should we do that?

Currently:
https://github.com/dask/dask-examples/blob/master/binder/environment.yml

channels:
  - conda-forge
dependencies:
  - python=3
  - bokeh=0.13
  - dask=0.20
  - dask-ml=0.10.0
  - distributed=1.24
  - jupyterlab=0.35.1
  - nodejs=8.9
  - numpy
  - pandas
  - pyarrow==0.10.0
  - scikit-learn=0.20
  - matplotlib
  - nbserverproxy
  - nomkl
  - h5py
  - xarray
  - bottleneck
  - py-xgboost
  - pip:
    - graphviz
    - dask_xgboost
    - seaborn
    - mimesis

[mrocklin]

Yes indeed. That would be good to do. Would you be willing to submit the PR? I'll be curious (and hopeful) to see if the tests pass or if there were any conflicting API changes.

…

On Tue, Feb 26, 2019 at 5:37 PM Genevieve Buckley ***@***.***> wrote: I noticed that the mybinder environment.yml file pins dask to version 0.20, but the latest dask release is now up to 1.1.2. It's probably time to update or unpin some of these dependencies. Should we do that? Currently: https://github.com/dask/dask-examples/blob/master/binder/environment.yml channels: - conda-forge dependencies: - python=3 - bokeh=0.13 - dask=0.20 - dask-ml=0.10.0 - distributed=1.24 - jupyterlab=0.35.1 - nodejs=8.9 - numpy - pandas - pyarrow==0.10.0 - scikit-learn=0.20 - matplotlib - nbserverproxy - nomkl - h5py - xarray - bottleneck - py-xgboost - pip: - graphviz - dask_xgboost - seaborn - mimesis — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#62>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszOEFeKO-oLAJUPUmowaK0I2tJdyeks5vReFmgaJpZM4bTgQg> .

[GenevieveBuckley]

Sure, I can do that.

Do you prefer to have things unpinned if possible, or just increment to a newer release? We'll probably run into this problem again (outdated dependencies) before too long if we keep things pinned, so I'd prefer not to. But I'd understand if you feel pull requests aren't frequent enough on this repo to trigger tests and alert us of any future issues.

[mrocklin]

I'd generally prefer that we keep things pinned. Otherwise bugs from mismatched versions might creep in without us knowing about it and things will go down and cause confusion. Instead I'd rather that we be explicit when we change versions and address any bugs that come up when we make such a change.

…

On Tue, Feb 26, 2019 at 6:32 PM Genevieve Buckley ***@***.***> wrote: Sure, I can do that. Do you prefer to have things unpinned if possible, or just increment to a newer release? We'll probably run into this problem again (outdated dependencies) before too long if we keep things pinned, so I'd prefer not to. But I'd understand if you feel pull requests aren't frequent enough on this repo to trigger tests and alert us of any future issues. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#62 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszBZs3AIkKVCj46HSjz_4Z7XbNS89ks5vRe5AgaJpZM4bTgQg> .

[GenevieveBuckley]

To update: I think it's better to put this issue on hold until the next dask-ml release is available.

I think this because upgrading dask versions has unearthed a few bugs, and the new release should contain some of the fixes we need.

Error 1

AttributeError: module 'dask' has no attribute 'sharedict', seen in the dask-examples/machine-learning/incremental.ipynb notebook. This is addressed by the merged PR dask/dask-ml#455,

Error 2

ValueError: high is out of bounds for int32, `, seen in the dask-examples/machine-learning.ipynb notebook. I think the open pull request dask/dask-ml#462 is likely to resolve this.

Error 3

I found that dask dataframe .std() produces error in cases where there are NaNs in output. Seen in the dask-examples/dataframes.ipynb notebook.
I raised an issue for it here: dask/dask#4534, and the associated pull request is here: dask/dask#4535
We can sidestep it entirely by replacing the dask dataframe std() example with mean() instead. I've changed that in the example notebook, so we don't have to wait for this bugfix to be released.

Other potential issues ahead

I also want to update the version of bokeh. Currently it's at 0.13 and the latest is >= 1.0.0 (a stable version 1.1.0 seems pretty close to release). I anticipate this will mean we'll have to update the examples using bokeh in the notebooks, as there are quite a few changes here.

CI build logs

My branch is here: https://github.com/GenevieveBuckley/dask-examples/tree/update-binder-env
You can take a look at the CI build logs here: https://travis-ci.com/GenevieveBuckley/dask-examples/branches

[mrocklin]

cc @TomAugspurger @jrbourbeau no pressure, but do you all have a timeline on a dask-ml release?

…

On Wed, Feb 27, 2019 at 11:18 PM Genevieve Buckley ***@***.***> wrote: To update: I think it's better to put this issue on hold until the next dask-ml release is available. I think this because upgrading dask versions has unearthed a few bugs, and the new release should contain some of the fixes we need. Error 1 AttributeError: module 'dask' has no attribute 'sharedict', seen in the dask-examples/machine-learning/incremental.ipynb notebook. This is addressed by the merged PR dask/dask-ml#455 <dask/dask-ml#455>, Error 2 ValueError: high is out of bounds for int32, `, seen in the dask-examples/machine-learning.ipynb notebook. I think the open pull request dask/dask-ml#462 <dask/dask-ml#462> is likely to resolve this. Error 3 I found that dask dataframe .std() produces error in cases where there are NaNs in output. Seen in the dask-examples/dataframes.ipynb notebook. I raised an issue for it here: dask/dask#4534 <dask/dask#4534>, and the associated pull request is here: dask/dask#4535 <dask/dask#4535> We can sidestep it entirely by replacing the dask dataframe std() example with mean() instead. I've changed that in the example notebook, so we don't have to wait for this bugfix to be released. Other potential issues ahead I also want to update the version of bokeh. Currently it's at 0.13 and the latest is >= 1.0.0 (a stable version 1.1.0 seems pretty close to release). I anticipate this will mean we'll have to update the examples using bokeh in the notebooks, as there are quite a few changes here. CI build logs My branch is here: https://github.com/GenevieveBuckley/dask-examples/tree/update-binder-env You can take a look at the CI build logs here: https://travis-ci.com/GenevieveBuckley/dask-examples/branches — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#62 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszCTCP2uIF-ahP2r0Ec_sePnLkcpoks5vR4KogaJpZM4bTgQg> .

[mrocklin]

Also, thanks for driving this and unearthing these bugs @GenevieveBuckley ! It's a great help.

[TomAugspurger]

Whoops forgot to last week. I’ll take a look at doing one today.

…

On Feb 28, 2019, at 09:15, Matthew Rocklin ***@***.***> wrote: cc @TomAugspurger @jrbourbeau no pressure, but do you all have a timeline on a dask-ml release? On Wed, Feb 27, 2019 at 11:18 PM Genevieve Buckley ***@***.***> wrote: > To update: I think it's better to put this issue on hold until the next > dask-ml release is available. > > I think this because upgrading dask versions has unearthed a few bugs, and > the new release should contain some of the fixes we need. > Error 1 > > AttributeError: module 'dask' has no attribute 'sharedict', seen in the > dask-examples/machine-learning/incremental.ipynb notebook. This is > addressed by the merged PR dask/dask-ml#455 > <dask/dask-ml#455>, > Error 2 > > ValueError: high is out of bounds for int32, `, seen in the > dask-examples/machine-learning.ipynb notebook. I think the open pull > request dask/dask-ml#462 <dask/dask-ml#462> is > likely to resolve this. > Error 3 > > I found that dask dataframe .std() produces error in cases where there are > NaNs in output. Seen in the dask-examples/dataframes.ipynb notebook. > I raised an issue for it here: dask/dask#4534 > <dask/dask#4534>, and the associated pull > request is here: dask/dask#4535 <dask/dask#4535> > We can sidestep it entirely by replacing the dask dataframe std() example > with mean() instead. I've changed that in the example notebook, so we > don't have to wait for this bugfix to be released. > Other potential issues ahead > > I also want to update the version of bokeh. Currently it's at 0.13 and the > latest is >= 1.0.0 (a stable version 1.1.0 seems pretty close to release). > I anticipate this will mean we'll have to update the examples using bokeh > in the notebooks, as there are quite a few changes here. > CI build logs > > My branch is here: > https://github.com/GenevieveBuckley/dask-examples/tree/update-binder-env > You can take a look at the CI build logs here: > https://travis-ci.com/GenevieveBuckley/dask-examples/branches > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#62 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AASszCTCP2uIF-ahP2r0Ec_sePnLkcpoks5vR4KogaJpZM4bTgQg> > . > — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[TomAugspurger]

released dask-ml 0.12.0. Binaries should be showing up in https://anaconda.org/conda-forge/dask-ml over the next hour or two. On Thu, Feb 28, 2019 at 9:23 AM Tom Augspurger <tom.augspurger88@gmail.com> wrote:

…

Whoops forgot to last week. I’ll take a look at doing one today. On Feb 28, 2019, at 09:15, Matthew Rocklin ***@***.***> wrote: cc @TomAugspurger @jrbourbeau no pressure, but do you all have a timeline on a dask-ml release? On Wed, Feb 27, 2019 at 11:18 PM Genevieve Buckley < ***@***.***> wrote: > To update: I think it's better to put this issue on hold until the next > dask-ml release is available. > > I think this because upgrading dask versions has unearthed a few bugs, and > the new release should contain some of the fixes we need. > Error 1 > > AttributeError: module 'dask' has no attribute 'sharedict', seen in the > dask-examples/machine-learning/incremental.ipynb notebook. This is > addressed by the merged PR dask/dask-ml#455 > <dask/dask-ml#455>, > Error 2 > > ValueError: high is out of bounds for int32, `, seen in the > dask-examples/machine-learning.ipynb notebook. I think the open pull > request dask/dask-ml#462 <dask/dask-ml#462> is > likely to resolve this. > Error 3 > > I found that dask dataframe .std() produces error in cases where there are > NaNs in output. Seen in the dask-examples/dataframes.ipynb notebook. > I raised an issue for it here: dask/dask#4534 > <dask/dask#4534>, and the associated pull > request is here: dask/dask#4535 <dask/dask#4535> > We can sidestep it entirely by replacing the dask dataframe std() example > with mean() instead. I've changed that in the example notebook, so we > don't have to wait for this bugfix to be released. > Other potential issues ahead > > I also want to update the version of bokeh. Currently it's at 0.13 and the > latest is >= 1.0.0 (a stable version 1.1.0 seems pretty close to release). > I anticipate this will mean we'll have to update the examples using bokeh > in the notebooks, as there are quite a few changes here. > CI build logs > > My branch is here: > https://github.com/GenevieveBuckley/dask-examples/tree/update-binder-env > You can take a look at the CI build logs here: > https://travis-ci.com/GenevieveBuckley/dask-examples/branches > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#62 (comment) >, > or mute the thread > < https://github.com/notifications/unsubscribe-auth/AASszCTCP2uIF-ahP2r0Ec_sePnLkcpoks5vR4KogaJpZM4bTgQg > > . > — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#62 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIiVkSdWQ6Shk9LrrwsRVK95ZK9qrks5vR_J7gaJpZM4bTgQg> .

[mrocklin]

I've changed that in the example notebook, so we don't have to wait for this bugfix to be released.

If possible I'd like to revert this change. I'd like for us to avoid having the examples work around bugs. I'm more than happy to prioritize bugfixes that block this. (I think that the underlying issue has now been fixed, and a dask/dask release is imminent).

[GenevieveBuckley]

I've changed that in the example notebook, so we don't have to wait for this bugfix to be released.

If possible I'd like to revert this change. I'd like for us to avoid having the examples work around bugs. I'm more than happy to prioritize bugfixes that block this. (I think that the underlying issue has now been fixed, and a dask/dask release is imminent).

Totally fine, I can revert that change. I was not expecting everyone to be so quick with bugfixes & new releases!