Python 3.6.2 math library may be incorrectly linked under latest builds

[cmeyer]

In our software, on Linux we load Python DLLs using dlopen.

I'm getting the following error and it looks like it is due to the way Python 3.6.2 is built or linked under latest Miniconda, perhaps from the clang change.

ImportError: /home/cmeyer/miniconda3/lib/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so: undefined symbol: PyArg_ParseTupleAndKeywords

When I look at the latest Miniconda build, I see the following:

cmeyer@ubuntu-dev: $ ldd ~/miniconda3/lib/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 
	linux-vdso.so.1 =>  (0x00007ffc3d59f000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd224d3b000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd224971000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd224667000)
	/lib64/ld-linux-x86-64.so.2 (0x000055c752594000)

Under 3.6.1 I see:

cmeyer@ubuntu-dev: $ ldd ~/miniconda3/lib/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so 
	linux-vdso.so.1 =>  (0x00007ffee1100000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f89ea10f000)
	libpython3.6m.so.1.0 => /home/cmeyer/miniconda3/lib/python3.6/lib-dynload/../../libpython3.6m.so.1.0 (0x00007f89e9c08000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f89e99ea000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f89e9620000)
	/lib64/ld-linux-x86-64.so.2 (0x000055f15f241000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f89e941c000)
	libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f89e9218000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f89e9010000)

The cause of this error seems to be that math on the latest Miniconda build Python 3.6.2 doesn't link to libpython3.6m.so.1.0 as Miniconda did before with Python 3.6.1.

Here is how I switched back to Python 3.6.1.

cmeyer@ubuntu-dev: $ ~/miniconda3/bin/conda install python=3.6.1
Fetching package metadata ...........
Solving package specifications: .

Package plan for installation in environment /home/cmeyer/miniconda3:

The following packages will be DOWNGRADED:

    python:   3.6.2-hdfe5801_15 --> 3.6.1-2 
    readline: 7.0-hac23ff0_3    --> 6.2-2   
    sqlite:   3.20.1-h6d8b0f3_1 --> 3.13.0-0
    tk:       8.6.7-h5979e9b_1  --> 8.5.18-0

Proceed ([y]/n)? y

For reference, here is my environment.

cmeyer@ubuntu-dev: $ ~/miniconda3/bin/conda update --all
Fetching package metadata ...........
Solving package specifications: .

# All requested packages already installed.
# packages in environment at /home/cmeyer/miniconda3:
#
asn1crypto                0.22.0           py36h265ca7c_1  
ca-certificates           2017.08.26           h1d4fec5_0  
certifi                   2017.7.27.1      py36h8b7b77e_0  
cffi                      1.10.0           py36had8d393_1  
chardet                   3.0.4            py36h0f667ec_1  
conda                     4.3.27           py36h2866c0b_0  
conda-env                 2.6.0                h36134e3_1  
cryptography              2.0.3            py36ha225213_1  
h5py                      2.7.0            py36he81ebca_1  
hdf5                      1.10.1               hb0523eb_0  
idna                      2.6              py36h82fb2a8_1  
intel-openmp              2018.0.0             h15fc484_7  
libedit                   3.1                  heed3624_0  
libffi                    3.2.1                h4deb6c0_3  
libgcc-ng                 7.2.0                hcbc56d2_1  
libgfortran               3.0.0                         1  
libgfortran-ng            7.2.0                h6fcbd8e_1  
libstdcxx-ng              7.2.0                h24385c6_1  
mkl                       2018.0.0             hb491cac_4  
ncurses                   6.0                  h06874d7_1  
numpy                     1.13.1           py36h5bc529a_2  
openssl                   1.0.2l               h9d1a558_3  
packaging                 16.8             py36ha668100_1  
pip                       9.0.1            py36h8ec8b28_3  
pycosat                   0.6.2            py36h1a0ea17_1  
pycparser                 2.18             py36hf9f622e_1  
pyopenssl                 17.2.0           py36h5cc804b_0  
pyparsing                 2.2.0            py36hee85983_1  
pysocks                   1.6.7            py36hd97a5b1_1  
python                    3.6.2               hdfe5801_15  
pytz                      2017.2           py36hc2ccc2a_1  
readline                  7.0                  hac23ff0_3  
requests                  2.18.4           py36he2e5f8d_1  
ruamel_yaml               0.11.14          py36ha2fb22d_2  
scipy                     0.19.1           py36h9976243_3  
setuptools                36.5.0           py36he42e2e1_0  
six                       1.10.0           py36hcac75e4_1  
sqlite                    3.20.1               h6d8b0f3_1  
tk                        8.6.7                h5979e9b_1  
urllib3                   1.22             py36hbe7ace6_0  
wheel                     0.29.0           py36he7f4e38_1  
xz                        5.2.3                h2bcbf08_1  
yaml                      0.1.7                h96e3832_1  
zlib                      1.2.11               hfbfcf68_1  

[mingwandroid]

clang is not used on Linux, GCC is (though our own, very modern GCC).

The issue you have is not that math is incorrectly linked, instead, what you have run into is that our Python 3 executables are now statically linked (I didn't have time to do this for Python 2.7). As a concession to people who want to embed Python we also provide a shared library.

Since the Python modules get their Python symbols from the static executable they do not dynamically link to libpython.so anymore (this accounts for part of the speed benefit).

For your use case, when loading our Python shared library, you will need to pass the RTLD_GLOBAL flag. I have not tested this as I consider your use-case out of scope since we never made any guarantees about how we will link our Python, but I would be interested to know if this works for you.

To 'correctly' embed our Python 3.6, following the official procedure you should do the following:

conda install gxx_linux-64 # (that you need gxx and not just gcc is a bug I will fix)
x86_64-conda_cos6-linux-gnu-gcc $(python3.6-config --cflags) -c main.c
x86_64-conda_cos6-linux-gnu-gcc main.o $(python3.6-config --ldflags)

This will link statically to libpython.a. If you want to force linking to libpython.so, you can use the following procedure:

conda install gxx_linux-64 # (that you need gxx and not just gcc is a bug I will fix)
x86_64-conda_cos6-linux-gnu-gcc $(python3.6-config --cflags) -c main.c
x86_64-conda_cos6-linux-gnu-gcc main.o -L${CONDA_PREFIX}/lib -lpython3.6m -lpthread -ldl -lutil -lrt -lm

[cmeyer]

Thanks. I'll give this a try and let you know the results. If possible, can we leave this issue open until the end of next week?

[mingwandroid]

Sure thing. I generally prefer for the reporter to close issues anyway.

[cmeyer]

Good news! The RTLD_GLOBAL did the trick. Thanks for the helpful response.

[mingwandroid]

Excellent. I'm glad this use-case still works then.

[jianxinwang]

I'm having this problem when trying to run a R test for (RxODE). MY python version is 37 though:

ImportError: /projects/industry/epd/software/Anaconda/py37/lib/python3.7/lib-dynload/math.cpython-37m-x86_64-linux-gnu.so: undefined symbol: PyArg_Parse

Please help. Thanks