-
Notifications
You must be signed in to change notification settings - Fork 54
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Install and import from kerchunk
(not fsspec-reference-maker)
#324
Conversation
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
Status update:
Re: kerchunk, even though there is a Conda Forge feedstock available, I've kept it as a The minimal reproducer is to build and activate an environment from # kerchunk-plus-pynio.yml
name: kerchunk-plus-pynio
channels:
- conda-forge
dependencies:
- python=3.8
- kerchunk>=0.0.4
- pynio with
Then call
which replicates this line from the xarray pynio backend, and raises Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/miniconda3/envs/kerchunk-plus-pynio-conda/lib/python3.8/site-packages/PyNIO/Nio.py", line 83, in <module>
from _nio import *
ImportError: dlopen(/miniconda3/envs/kerchunk-plus-pynio-conda/lib/python3.8/site-packages/PyNIO/_nio.cpython-38-darwin.so, 0x0002): Library not loaded: @rpath/libtbb.dylib
Referenced from: /miniconda3/envs/kerchunk-plus-pynio-conda/lib/libtiledb.dylib
Reason: tried: '/miniconda3/envs/kerchunk-plus-pynio-conda/lib/libtbb.dylib' (no such file), '/miniconda3/envs/kerchunk-plus-pynio-conda/lib/libtbb.dylib' (no such file), '/miniconda3/envs/kerchunk-plus-pynio-conda/lib/libtbb.dylib' (no such file), '/miniconda3/envs/kerchunk-plus-pynio-conda/lib/libtbb.dylib' (no such file), '/miniconda3/envs/kerchunk-plus-pynio-conda/lib/libtbb.dylib' (no such file), '/miniconda3/envs/kerchunk-plus-pynio-conda/lib/libtbb.dylib' (no such file), '/miniconda3/envs/kerchunk-plus-pynio-conda/lib/libtbb.dylib' (no such file), '/miniconda3/envs/kerchunk-plus-pynio-conda/lib/python3.8/site-packages/PyNIO/../../../libtbb.dylib' (no such file), '/miniconda3/envs/kerchunk-plus-pynio-conda/lib/libtbb.dylib' (no such file), '/miniconda3/envs/kerchunk-plus-pynio-conda/lib/libtbb.dylib' (no such file), '/miniconda3/envs/kerchunk-plus-pynio-conda/lib/libtbb.dylib' (no such file), '/miniconda3/envs/kerchunk-plus-pynio-conda/lib/python3.8/site-packages/PyNIO/../../../libtbb.dylib' (no such file), '/miniconda3/envs/kerchunk-plus-pynio-conda/bin/../lib/libtbb.dylib' (no such file), '/miniconda3/envs/kerchunk-plus-pynio-conda/bin/../lib/libtbb.dylib' (no such file), '/usr/local/lib/libtbb.dylib' (no such file), '/usr/lib/libtbb.dylib' (no such file) Installing kerchunk from Is this worth raising on https://github.com/conda-forge/kerchunk-feedstock? I ask because the xarray docs indicate that pynio is no longer actively maintained so maybe chasing pynio compatibility is not a relevant concern, and the solution is for us to just drop the pynio dependency? |
Since the original fsspec-reference-maker surely has no users, I could remove it from PyPI, if that helps; but it's more involved fro conda-forge. I can also remove the alias package from kerchunk - but you won't notice that unless I make a new release (which would bring combine2 with it, since I should merge that). |
For a fix for the nbmake issue, see pydata/xarray#6350. |
@martindurant, this PR resolves our |
kerchunk's deps are extremely minimal, so I don't see how that's causing a problem. requirements:
host:
- pip
- python >=3.7
run:
- fsspec
- python >=3.7
- ujson |
Since Edit: I will test this and report back. |
Don't know, maybe? It says the lib was missing, though, not some symbol clash. For reference, libtbb is some multiprocessing abstraction. Could try explicitly listing ujson with some specific version, but that's shooting in the dark. |
Thanks Ryan. 4fa6fea also fixes this, because it bumps to |
Martin, is the removal of |
Yes, this is unavoidable. I am looking at this repo now for the fix. The signature is now:
where probably the only content of |
Re: name: ujson-pynio
channels:
- conda-forge
dependencies:
- python=3.8
- pynio
- ujson I'm a bit out of my depth in debugging this, but my hunch is that pynio being unmaintained is the problem, and maybe we want to just drop the pynio dependency? |
Certainly, ujson is common enough that it could easily creep into the environment via some other dependency in another package later, so if we don't need pynio... |
The |
Per offline conversation with @rabernat, I'll merge this as-is, with the knowledge that after the next |
I believe the fact that upstream-dev tests are failing, despite an upstream fix in kerchunk may be related to the fact that we never migrated from installing and importing from the the fsspec-reference-maker namespace.
As a result, when we install the upstream dev requirements, we get a kerchunk installation in addition to an fsspec-reference-maker installation. Obviously, this is problematic.
I hope migrating to the
kerchunk
namespace for installation and import may solve this.