Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Runtime error loading numpy with python 3.6.4 #116

Closed
SemanticBeeng opened this issue Mar 10, 2018 · 4 comments
Closed

Runtime error loading numpy with python 3.6.4 #116

SemanticBeeng opened this issue Mar 10, 2018 · 4 comments

Comments

@SemanticBeeng
Copy link

SemanticBeeng commented Mar 10, 2018 •
edited
Loading

I am using ScalaPy to load Numpy through Jep and getting a linkage error.

Exception in thread "main" jep.JepException: <class 'ImportError'>: /development/bin/python/conda3/lib/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so: undefined symbol: PyArg_ParseTupleAndKeywords
	at /development/bin/python/conda3/lib/python3.6/site-packages/numpy/lib/__init__.<module>(__init__.py:3)
	at /development/bin/python/conda3/lib/python3.6/site-packages/numpy/add_newdocs.<module>(add_newdocs.py:13)
	at /development/bin/python/conda3/lib/python3.6/site-packages/numpy/__init__.<module>(__init__.py:142)
	at <string>.<module>(<string>:1)
	at jep.Jep.eval(Native Method)
	at jep.Jep.eval(Jep.java:609)
	at me.shadaj.scalapy.py.Module$.apply(Module.scala:48)
	at me.shadaj.scalapy.py.package$.module(package.scala:10)
	at me.shadaj.scalapy.tensorflow.Example$.delayedEndpoint$me$shadaj$scalapy$tensorflow$Example$1(Example.scala:15)
	at me.shadaj.scalapy.tensorflow.Example$delayedInit$body.apply(Example.scala:7)

This other thing seemed related - "concession to people who want to embed Python we also provide a shared library." - but cannot tell for sure
ContinuumIO/anaconda-issues#6401 (comment)

My Python environment

$ conda list
# packages in environment at /development/bin/python/conda3:
#
# Name                    Version                   Build  Channel
absl-py                   0.1.10                   py36_0
asn1crypto                0.24.0                   py36_0
binutils_impl_linux-64    2.28.1               had2808c_3
binutils_linux-64         7.2.0                        25
bleach                    1.5.0                    py36_0    anaconda
ca-certificates           2017.08.26           h1d4fec5_0
certifi                   2018.1.18                py36_0
cffi                      1.11.5           py36h9745a5d_0
chardet                   3.0.4            py36h0f667ec_1
conda                     4.4.11                   py36_0
conda-env                 2.6.0                h36134e3_1
cryptography              2.1.4            py36hd09be54_0
gcc_impl_linux-64         7.2.0                habb00fd_3
gcc_linux-64              7.2.0                        25
gxx_impl_linux-64         7.2.0                hdf63c60_3
gxx_linux-64              7.2.0                        25
html5lib                  0.9999999                py36_0    anaconda
idna                      2.6              py36h82fb2a8_1
intel-openmp              2018.0.0             hc7b2577_8
jep                       3.7.1                     <pip>
krb5                      1.14.2               hcdc1b81_6
libedit                   3.1                  heed3624_0
libffi                    3.2.1                hd88cf55_4
libgcc                    7.2.0                h69d50b8_2
libgcc-ng                 7.2.0                hdf63c60_3
libgfortran-ng            7.2.0                hdf63c60_3
libpq                     9.6.6                h4e02ad2_0
libprotobuf               3.5.1                h6f1eeef_0
libstdcxx-ng              7.2.0                hdf63c60_3
markdown                  2.6.11                   py36_0
mkl                       2018.0.1             h19d6760_4
mypy                      0.550            py36hd8b1e0f_0
mypy                      0.520                     <pip>
ncurses                   6.0                  h9df7e31_2
nltk                      3.2.5            py36h7532b22_0
numpy                     1.14.1           py36ha266831_2
openssl                   1.0.2n               hb7f436b_0
packaging                 17.1                     py36_0
pip                       9.0.1                    py36_5
protobuf                  3.5.1            py36hf484d3e_0
psutil                    5.4.3            py36h14c3975_0
psycopg2                  2.7.4            py36hb7f436b_0
pycallgraph               1.0.1                     <pip>
pycosat                   0.6.3            py36h0a5515d_0
pycparser                 2.18             py36hf9f622e_1
pyopenssl                 17.5.0           py36h20ba746_0
pyparsing                 2.2.0            py36hee85983_1
pysocks                   1.6.8                    py36_0
python                    3.6.4                hc3d631a_1
readline                  7.0                  ha6073c6_4
requests                  2.18.4           py36he2e5f8d_1
ruamel_yaml               0.15.35          py36h14c3975_1
setuptools                38.5.1                   py36_0
six                       1.11.0           py36h372c433_1
sqlite                    3.22.0               h1bed415_0
tensorflow                1.5.0                         0
tensorflow-base           1.5.0            py36hff88cb2_1
tensorflow-tensorboard    1.5.1            py36hf484d3e_0
tk                        8.6.7                hc745277_3
typed-ast                 1.1.0            py36h4769eee_0
typed-ast                 1.0.4                     <pip>
urllib3                   1.22             py36hbe7ace6_0
werkzeug                  0.14.1                   py36_0
wheel                     0.30.0           py36hfd4bba0_1
xz                        5.2.3                h55aa19d_2
yaml                      0.1.7                had09818_2
zlib                      1.2.11               ha838bed_2

Code to reproduce can be found here
SemanticBeeng/scalapy-tensorflow@e1922d3

Please be kind to advise
@bsteffensmeier
Copy link
Member

Thanks for the link to the anaconda issue, it is most definitely related. Starting in version 3.8(not yet released) we try to reload libpython with RTLD_GLOBAL for people who build python libraries without declaring dependencies. It would be helpful if you could check out the dev_3.8 branch and see if that resolves your issue.

For the current version of jep the only workaround is to set the LD_PRELOAD environmental variable to the location of libpython, there is more information on this in the Linux section of the wiki.

@SemanticBeeng
Copy link
Author

SemanticBeeng commented Mar 10, 2018 •
edited
Loading

Thank you for following up.

Tried LD_PRELOAD with libraries from this list

$ find /development/bin/python/conda3/envs/py3/ -name libpython*.*
/development/bin/python/conda3/envs/py3/lib/python3.6/config-3.6m-x86_64-linux-gnu/libpython3.6m.a
/development/bin/python/conda3/envs/py3/lib/libpython3.6m.so.1.0
/development/bin/python/conda3/envs/py3/lib/libpython3.6m.so
/development/bin/python/conda3/envs/py3/lib/libpython3.so

and now getting something similar but not about the math package

xception in thread "main" jep.JepException: <class 'ImportError'>: /development/bin/python/conda3/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _PyUnicode_AsWideCharString
	at /development/bin/python/conda3/lib/python3.6/ctypes/__init__.<module>(__init__.py:7)
	at /development/bin/python/conda3/lib/python3.6/site-packages/numpy/_distributor_init.__enter__(_distributor_init.py:18)
	at /development/bin/python/conda3/lib/python3.6/site-packages/numpy/_distributor_init.<module>(_distributor_init.py:33)
	at /development/bin/python/conda3/lib/python3.6/site-packages/numpy/__init__.<module>(__init__.py:156)

Did not yet try the 3.8 branch.

So this LD_PRELOAD way is supposed to work or may or may not work?
Just trying to understand if I am not using it correctly?

Tried to see if something wrong with ctypes and followed https://stackoverflow.com/a/35158591/4032515

For me python3 -c 'import ctypes' does not generate an error.

@SemanticBeeng
Copy link
Author

My same code seems to work fine with a fresh python 3.5 virtual env

Should we conclude that 3.6 has improperly linked binaries ? Or my python env has some mixed up libraries ?

Used and LD_PRELOAD is strictly necessary.

-Djava.library.path=/development/bin/python/conda3/envs/py35/lib/python3.5/site-packages/jep
PYTHONHOME=/development/bin/python/conda3/envs/py35/
LD_PRELOAD=/development/bin/python/conda3/envs/py35/lib/libpython3.5m.so.1.0

$ conda list
# packages in environment at /development/bin/python/conda3/envs/py35:
#
# Name                    Version                   Build  Channel
absl-py                   0.1.10                   py35_0    anaconda
alabaster                 0.7.10           py35h6fb19ab_0  
anaconda                  custom           py35h104c396_0    anaconda
anaconda-client           1.6.9                    py35_0  
anaconda-project          0.8.2            py35h7d62c80_0  
asn1crypto                0.24.0                   py35_0  
astroid                   1.6.1                    py35_0  
astropy                   2.0.3            py35h14c3975_0  
attrs                     17.4.0                   py35_0  
babel                     2.5.3                    py35_0  
backports                 1.0              py35hd471ac7_1  
backports.shutil_get_terminal_size 1.0.0            py35h40844db_2  
beautifulsoup4            4.6.0            py35h442a8c9_1  
bitarray                  0.8.1            py35h14c3975_1  
bkcharts                  0.2              py35he4f7e30_0  
blaze                     0.11.3           py35ha2c9137_0  
bleach                    1.5.0                    py35_0    anaconda
bokeh                     0.12.13          py35h2f9c1c0_0  
boto                      2.48.0           py35h2cfd601_1  
bottleneck                1.2.1            py35he1b16f1_0  
bzip2                     1.0.6                h9a117a8_4  
ca-certificates           2017.08.26           h1d4fec5_0    anaconda
cairo                     1.14.12              h77bcde2_0  
certifi                   2018.1.18                py35_0    anaconda
cffi                      1.11.4           py35h9745a5d_0  
chardet                   3.0.4            py35hb6e9ddf_1  
click                     6.7              py35h353a69f_0  
cloudpickle               0.5.2                    py35_1  
clyent                    1.2.2            py35h491ffcb_1  
colorama                  0.3.9            py35h81e2b6c_0  
contextlib2               0.5.5            py35h6690dba_0  
cryptography              2.1.4            py35hbeb2da1_0  
curl                      7.58.0               h84994c4_0  
cycler                    0.10.0           py35hc4d5149_0  
cython                    0.27.3           py35h6cdc64b_0  
cytoolz                   0.9.0            py35h14c3975_0  
dask                      0.16.1                   py35_0  
dask-core                 0.16.1                   py35_0  
datashape                 0.5.4            py35h5b068ae_0  
dbus                      1.12.2               hc3f9b76_1  
decorator                 4.2.1                    py35_0  
distributed               1.20.2                   py35_0  
docutils                  0.14             py35hd11081d_0  
entrypoints               0.2.3            py35h48174a2_2  
et_xmlfile                1.0.1            py35hde8d43e_0  
expat                     2.2.5                he0dffb1_0  
fastcache                 1.0.2            py35h14c3975_2  
filelock                  2.0.13           py35h4160c53_0  
flask                     0.12.2           py35h679e90e_0  
flask-cors                3.0.3            py35h9b2065d_0  
fontconfig                2.12.4               h88586e7_1  
freetype                  2.8                  hab7d2ae_1  
get_terminal_size         1.0.0                haa9412d_0  
gevent                    1.2.2            py35he064abf_0  
glib                      2.53.6               h5d9569c_2  
glob2                     0.6              py35hff66265_0  
gmp                       6.1.2                h6c8ec71_1  
gmpy2                     2.0.8            py35hd0a1c9a_2  
graphite2                 1.3.10               hf63cedd_1  
greenlet                  0.4.12           py35h2547b41_0  
gst-plugins-base          1.12.4               h33fb286_0  
gstreamer                 1.12.4               hb53b477_0  
h5py                      2.7.1            py35h8d53cdc_0  
harfbuzz                  1.7.4                hc5b324e_0  
hdf5                      1.10.1               h9caa474_1  
heapdict                  1.0.0                    py35_2  
html5lib                  0.9999999                py35_0    anaconda
icu                       58.2                 h9c2bf20_1  
idna                      2.6              py35h8605a33_1  
imageio                   2.2.0            py35hd0a6de2_0  
imagesize                 0.7.1            py35hf008fae_0  
intel-openmp              2018.0.0             hc7b2577_8  
ipykernel                 4.8.0                    py35_0  
ipython                   6.2.1            py35hd850d2a_1  
ipython_genutils          0.2.0            py35hc9e07d0_0  
ipywidgets                7.1.1                    py35_0  
isort                     4.2.15           py35h44a00d2_0  
itsdangerous              0.24             py35h7c46880_1  
jbig                      2.1                  hdba287a_0  
jdcal                     1.3              py35hf1f7175_0  
jedi                      0.11.1                   py35_0  
jep                       3.7.1                     <pip>
jinja2                    2.10             py35h480ab6d_0  
jpeg                      9b                   h024ee3a_2  
jsonschema                2.6.0            py35h4395190_0  
jupyter                   1.0.0                    py35_4  
jupyter_client            5.2.2                    py35_0  
jupyter_console           5.2.0            py35h4044a63_1  
jupyter_core              4.4.0            py35ha89e94b_0  
jupyterlab                0.31.5                   py35_0  
jupyterlab_launcher       0.10.2                   py35_0  
lazy-object-proxy         1.3.1            py35h4c720c6_0  
libcurl                   7.58.0               h1ad7b7a_0  
libedit                   3.1                  heed3624_0  
libffi                    3.2.1                hd88cf55_4  
libgcc-ng                 7.2.0                h7cc24e2_2  
libgfortran-ng            7.2.0                h9f7466a_2  
libpng                    1.6.34               hb9fc6fc_0  
libprotobuf               3.5.1                h6f1eeef_0    anaconda
libsodium                 1.0.15               hf101ebd_0  
libssh2                   1.8.0                h9cfc8f7_4  
libstdcxx-ng              7.2.0                h7a57d05_2  
libtiff                   4.0.9                h28f6b97_0  
libtool                   2.4.6                h544aabb_3  
libxcb                    1.12                 hcd93eb1_4  
libxml2                   2.9.7                h26e45fe_0  
libxslt                   1.1.32               h1312cb7_0  
llvmlite                  0.21.0           py35hcfd7307_0  
locket                    0.2.0            py35h170bc82_1  
lxml                      4.1.1            py35hf71bdeb_1  
lzo                       2.10                 h49e0be7_2  
markdown                  2.6.11                   py35_0    anaconda
markupsafe                1.0              py35h4f4fcf6_1  
matplotlib                2.1.2            py35h0e671d2_0  
mccabe                    0.6.1            py35hd8c1f23_1  
mistune                   0.8.3                    py35_0  
mkl                       2018.0.1             h19d6760_4  
mkl-service               1.1.2            py35h0fc7090_4  
mpc                       1.0.3                hec55b23_5  
mpfr                      3.1.5                h11a74b3_2  
mpmath                    1.0.0            py35h7ce6e34_2  
msgpack-python            0.5.1            py35h6bb024c_0  
multipledispatch          0.4.9            py35h2ff591a_0  
nbconvert                 5.3.1            py35hc5194e3_0  
nbformat                  4.4.0            py35h12e6e07_0  
ncurses                   6.0                  h9df7e31_2  
networkx                  2.1                      py35_0  
nltk                      3.2.5            py35h09ad193_0  
nose                      1.3.7            py35hdc64897_2  
notebook                  5.4.0                    py35_0  
numba                     0.36.2          np114py35ha126a20_0  
numexpr                   2.6.4            py35h119f745_0  
numpy                     1.14.1           py35ha266831_2    anaconda
numpydoc                  0.7.0            py35he089a57_0  
odo                       0.5.1            py35h102315f_0  
olefile                   0.45.1                   py35_0  
openpyxl                  2.4.10                   py35_0  
openssl                   1.0.2n               hb7f436b_0    anaconda
packaging                 16.8             py35h2260b46_1  
pandas                    0.22.0           py35hf484d3e_0  
pandoc                    1.19.2.1             hea2e7c5_1  
pandocfilters             1.4.2            py35h1565a15_1  
pango                     1.41.0               hd475d92_0  
parso                     0.1.1            py35h1b200a3_0  
partd                     0.3.8            py35h68187f2_0  
patchelf                  0.9                  hf79760b_2  
path.py                   10.5             py35h39c98c1_0  
pathlib2                  2.3.0            py35hd637de4_0  
patsy                     0.5.0                    py35_0  
pcre                      8.41                 hc27e229_1  
pep8                      1.7.1                    py35_0  
pexpect                   4.3.1                    py35_0  
pickleshare               0.7.4            py35hd57304d_0  
pillow                    5.0.0            py35h3deb7b8_0  
pip                       9.0.1            py35h7e7da9d_4  
pixman                    0.34.0               hceecf20_3  
pkginfo                   1.4.1            py35hb526c55_1  
pluggy                    0.6.0            py35hb428803_0  
ply                       3.10             py35h459f1a0_0  
prompt_toolkit            1.0.15           py35hc09de7a_0  
protobuf                  3.5.1            py35hf484d3e_0    anaconda
psutil                    5.4.3            py35h14c3975_0  
ptyprocess                0.5.2            py35h38ce0a3_0  
py                        1.5.2            py35hfe8c5ce_0  
pycodestyle               2.3.1            py35h8d2105a_0  
pycosat                   0.6.3            py35h6b6bb97_0  
pycparser                 2.18             py35h61b3040_1  
pycrypto                  2.6.1            py35h14c3975_7  
pycurl                    7.43.0.1         py35hb7f436b_0  
pyflakes                  1.6.0            py35hd1da72e_0  
pygments                  2.2.0            py35h0f41973_0  
pylint                    1.8.2                    py35_0  
pyodbc                    4.0.22           py35hf484d3e_0  
pyopenssl                 17.5.0           py35h4f8b8c8_0  
pyparsing                 2.2.0            py35h041ed72_1  
pyqt                      5.6.0            py35h0e41ada_5  
pysocks                   1.6.7            py35h6aefbb0_1  
pytables                  3.4.2            py35hfa98db7_2  
pytest                    3.3.2                    py35_0  
python                    3.5.4               h417fded_24  
python-dateutil           2.6.1            py35h90d5b31_1  
pytz                      2017.3           py35hb13c558_0  
pywavelets                0.5.2            py35h53ec731_0  
pyyaml                    3.12             py35h46ef4ae_1  
pyzmq                     16.0.3           py35ha889422_0  
qt                        5.6.2               h974d657_12  
qtawesome                 0.4.4            py35hd68ff89_0  
qtconsole                 4.3.1            py35h4626a06_0  
qtpy                      1.3.1            py35h881af26_0  
readline                  7.0                  ha6073c6_4  
requests                  2.18.4           py35hb9e6ad1_1  
rope                      0.10.7           py35h986144c_0  
ruamel_yaml               0.15.35          py35h14c3975_1  
scikit-image              0.13.1           py35h14c3975_1  
scikit-learn              0.19.1           py35hbf1f462_0  
scipy                     1.0.0            py35hcbbe4a2_0  
seaborn                   0.8.1            py35h04cba02_0  
send2trash                1.4.2                    py35_0  
setuptools                38.4.0                   py35_0  
simplegeneric             0.8.1                    py35_2  
singledispatch            3.4.0.3          py35h0cd4ec3_0  
sip                       4.18.1           py35h9eaea60_2  
six                       1.11.0           py35h423b573_1  
snowballstemmer           1.2.1            py35h5435977_0  
sortedcollections         0.5.3            py35hb2f60ff_0  
sortedcontainers          1.5.9                    py35_0  
sphinx                    1.6.6                    py35_0  
sphinxcontrib             1.0              py35hcb7efd2_1  
sphinxcontrib-websupport  1.0.1            py35h3537809_1  
spyder                    3.2.6                    py35_0  
sqlalchemy                1.2.1            py35h14c3975_0  
sqlite                    3.22.0               h1bed415_0  
statsmodels               0.8.0            py35haa9d50b_0  
sympy                     1.1.1            py35h919b29a_0  
tblib                     1.3.2            py35hf1eb0b4_0  
tensorflow                1.5.0                         0    anaconda
tensorflow-base           1.5.0            py35hff88cb2_1    anaconda
tensorflow-tensorboard    1.5.1            py35hf484d3e_0    anaconda
terminado                 0.8.1                    py35_1  
testpath                  0.3.1            py35had42eaf_0  
tk                        8.6.7                hc745277_3  
toolz                     0.9.0                    py35_0  
tornado                   4.5.3                    py35_0  
traitlets                 4.3.2            py35ha522a97_0  
typing                    3.6.2            py35hcadae7e_0  
unicodecsv                0.14.1           py35h4805c0c_0  
unixodbc                  2.3.4                hc36303a_1  
urllib3                   1.22             py35h2ab6e29_0  
wcwidth                   0.1.7            py35hcd08066_0  
webencodings              0.5.1            py35hb6cf162_1  
werkzeug                  0.14.1                   py35_0  
wheel                     0.30.0           py35hd3883cf_1  
widgetsnbextension        3.1.0                    py35_0  
wrapt                     1.10.11          py35hfdafd39_0  
xlrd                      1.1.0            py35h45a0a2a_1  
xlsxwriter                1.0.2            py35h8ee9321_0  
xlwt                      1.3.0            py35h884a0cf_0  
xz                        5.2.3                h55aa19d_2  
yaml                      0.1.7                had09818_2  
zeromq                    4.2.2                hbedb6e5_2  
zict                      0.1.3            py35h29275ca_0  
zlib                      1.2.11               ha838bed_2

Will close since does not seem to be an issue with jep itself.
Thanks again.

SemanticBeeng added a commit to SemanticBeeng/scalapy-tensorflow that referenced this issue Mar 10, 2018
@SemanticBeeng
#fix Had an runtime error with my 3.6 virtual env but seems to work…
b882894 
… with a fresh 3.5

ninia/jep#116 (comment)

Using these from the Intellij debug configuration

```
-Djava.library.path=/development/bin/python/conda3/envs/py35/lib/python3.5/site-packages/jep
PYTHONHOME=/development/bin/python/conda3/envs/py35/
LD_PRELOAD=/development/bin/python/conda3/envs/py35/lib/libpython3.5m.so.1.0
```

Also getting this but just a warning
https://stackoverflow.com/a/47227886/4032515
@ranglang
Copy link

same error

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@SemanticBeeng @ranglang @bsteffensmeier