-
-
Notifications
You must be signed in to change notification settings - Fork 13
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remove optional dependencies #5
Comments
I got a request from AOS users to do that and ensure that everything "just works." But I understand that there is an extra burden on the network to download all those packages. Unfortunately |
As long as these dependencies were intentionally added I have no issue with them being there. Just wanted to make sure this was not included accidentally since they do differ from the requirements of the package in |
Sometimes I do worry that the extra dependencies might cause the solver to prefer an older version of |
It is a bit confusing, particularly since
I do understand that. The other side, in production projects a too large number of uncontrolled dependencies can be reason enough not to use a package.. For instance in an environment with numpy, scipy, and pandas installed, conda create -c conda-forge -n test-env numpy scipy pandas python=3.6
source activate test-env xarray from the Interestingly, matplotlib, while it can be used for plotting xarrays, doesn't get pulled, wheras bokeh does.. Maybe at least it could worthwhile to depend on the recently created dask-core package instead of the dask metapackage? So that bokeh etc. wouldn't be pulled when one installs xarray.. Sorry for the rant, xarray / conda-forge are great... I can do a PR to help.. Full list of packages below
|
@rth I guess that you can still use I can say for sure that 99% of the users of the conda-forge However, I'll defer to the rest of the @conda-forge/xarray team here, but I am 👎 to remove |
@ocefpaf Thanks for the detailed explanations! Fair enough, from the perspective of users who install xarray directly I understand. My issue is with how this fits into the wider ecosystem. Say one works on some scientific package that has ~10 dependencies in a
but then this would pull all the dependencies mentioned above due to xarray. This is problematic for several reasons,
To fix it I can certainly play with the conda channel priorities (good luck making sure this survives on user's PCs), or split the install in two etc. neither of which is very satisfactory. Another issue is that there is no way of knowing that the conda-forge version of xarray is designed for Met/Ocean community (short of talking to the maintainers) or to what extent this may change in the future. For my personal use of xarray, this is not an actual issue at the moment. I'm also aware that there might not be much that could be done about this due to the way conda packaging works, so it's a more a general comment. Sorry for the long response. |
Indeed that is not in the docs 😄 but conda-forge was born from the merge of the UK MetOffice and the IOOS conda channels, so it is deeply rooted in the Met/Ocean community 😉
The future is an unknown. Since the creation of conda-forge we have many other communities joined us, like bioconda and their r-packages. However, like I said before, even though I am 👎 I do defer this to the team here in case they want to change the current policy. With that said conda is bringing the concept of optional packages soon we can wait for that. Another alternative would be a
No problem. I welcome this kind of discussion, otherwise we won't evolve. |
I wonder if we should/could setup a few metapackages ( |
Would it make sense to create an This is the layout that the |
@jjhelmus that is what I suggested above. But like I mentioned I am not a big fan of 'expanding the namespace'. |
I'm still a little surprised that conda doesn't have a notion of "optional
dependencies" like pip/setuptools.
…On Wed, Sep 20, 2017 at 3:34 PM Filipe ***@***.***> wrote:
@jjhelmus <https://github.com/jjhelmus> that is what I suggested above.
But like I mentioned I am not a big fan of 'expanding the namespace'.
—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub
<#5 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABKS1oFV9vJjJX9xrz2Z0N83cUNLdTx6ks5skZL9gaJpZM4JQJAb>
.
|
@ocefpaf Thanks for the interesting discussion!
Aww, I had no idea. Makes sense now )
Well, there is a PR conda/conda#4982 merged in conda 4.4, but I'm still not able to find good documentation about it. As far as I understood it a different approach than the square bracket notation on pypi conda/conda#3299 (comment) . For instance for dask that didn't seem to work conda-forge/dask-feedstock#22 (comment) hence the reason for going with a meta-package. I agree that cluttering the namespace with multiple names for the same package might not be the solution. Particularly since it would mean that |
👍 to that! @rth I'd rather break the current setup than going for meta-packages.
If that is OK for you I'd rather wait as well. Note that I am defending my community here, but if people can make a strong argument that removing the dependencies would be better for a larger community I will comply (with some complaining along the way 😉). |
For future reference, assuming that h5netcdf is (or will be one day) stable enough, feature complete (with repect to xarray needs) and that xarray will automatically switch to the available netcdf backend, it might be worth considering using it as dependency instead of netcdf. h5netcdf conda package only depends on |
We came across this with @jasongrout and we would be in favor of dropping optional dependencies. Dask and netcdf are pretty heavy dependencies. For a project like |
I'm leaning towards an |
Wouldn't that be weird to depart from the pure python package naming? |
That is one of the reasons I don't really like that solution, the B/c of that I'm more in favor of shipping |
If you have constraints on the versions of dask and netcdf that play well with xarray, you can use |
Otherwise, I would lean towards dropping completely unneeded dependencies, and let users install them via another meta package or simply in their environment. |
BTW @SylvainCorlay I looked into the velocity plugin in (But that is an issue for |
Yep! Thanks for looking into it! Although that is a bit orthogonal to |
Actually, the arguments at the ipyleaflet discussion probably apply here as well. Meta-issue: do you include dependencies because you think that lots of users will use it, or because it's actually needed? Same question for both repos :). |
Well, ipyleaflet actually uses xarray. There is a question about whether we should use it. (Not the same question.) |
And the question was triggered by that thing pulling too many dependencies. |
@ocefpaf's point is that it's only using xarray because it's convenient (i.e., lots of users will use it that way) - and that xarray is not actually needed - that's the connection I saw. But digression aside, both specific questions have good points. |
Let's keep thinking about this. I want @shoyer to weigh in as well. |
@jhamman note that everything I said here is my personal opinion and not a |
@ocefpaf. Thanks. Remember, you're on the @conda-forge/xarray team too! |
My general opinion is that it's best to restrict explicit dependencies to strict requirements. This retains maximum flexibility, at a slight cost of convenience. Optional dependencies can be put in new metapackages, e.g., "xarray[all]" or however you spell that with conda. |
Okay, I expect this will cause some pain downstream but I'm okay with this plan. Should we time this with the upcoming minor release (xarray v0.11)? |
That is the problem, you don't 😄
It is not too bad actually. People will have to install PS: @SylvainCorlay note that this will break the example in |
Also, as far as I remember |
Yes, the binder will need netcdf indeed, although probably not the ipyleaflet package. |
I would agree here.
Why is xarray-extras problematic? Presumably it would be a simple package that depends on xarray, dask, netCDF4, etc. without providing any packages itself. This makes it easy to install the entire xarray stack, e.g., with |
B/c it breaks the 1-1 correspondence with PyPI and sounds more like a forkish behavior from the conda community. (We are always criticized for that.). I'd rather have a single |
I think Ideally, of course, something like |
I agree with you, the rest of the Python community that complains to me at events when I talk about conda/conda-forge does not.
Yes, that is the dream. I know it is on their road-map but not sure when this feature would land on the conda world. (Wink, wink at @kalefranz again.) |
I think it is possible to have:
The square bracket naming convention is unfortunately impossible with conda it seems due to fancy version specifications, although maybe we can use a The 3 points can be achieved with a combination of the A sample recipe can be found here for dask. Xref: Dask issue: conda-forge/dask-feedstock#46 |
I noticed that a number of optional dependencies (scipy, dask, etc) are being listed in the requirement section of this recipe, can these be removed? Does
conda-forge
have a recommendation on including optional dependencies in a recipe?The text was updated successfully, but these errors were encountered: