Bootstrap fails using --use-conda on MacOS

[luc-j-bourhis]

Here are the steps to reproduce the problem.

% micromamba create -n cctbx-dev python=3.10
% python bootstrap.py --builder cctbxlite --nproc=4 --git-ssh --use-conda ~/micromamba/envs/cctbx-dev

I used micromamba instead of conda but I see no reason it should matter. Apart from the path ~/micromamba/envs/cctbx-dev being different. This fails:

===== Running in build: run configure.py
Traceback (most recent call last):
  File "/Users/luc/Developer/bootstrap.py", line 180, in run
    p = subprocess.Popen(
        ^^^^^^^^^^^^^^^^^
  File "/Users/luc/.pyenv/versions/3.12.2/lib/python3.12/subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/Users/luc/.pyenv/versions/3.12.2/lib/python3.12/subprocess.py", line 1953, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/luc/micromamba/envs/cctbx-dev/python.app/Contents/MacOS/python'

It looks like boostrap.py looks for a typical MacOS Python executable inside the conda environment, but it will never be the case, will it?

[luc-j-bourhis]

I see: I need to install anaconda::python.app in the conda environment, not just python. But then that environment also needs packages future, six, and scons. Fair enough, --use-conda is an advanced option but I suggest documenting a bit more what it entails.

[luc-j-bourhis]

Then a specific version of python.app is necessary, as otherwise the compilation fail inside a Boost.Python header. Eventually, this is what worked:

micromamba create -n cctbx-dev -c conda-forge python.app=3=py39hca72f7f_0 future six scons
python bootstrap.py --builder cctbxlite --nproc=4 --git-ssh --use-conda ~/micromamba/envs/cctbx-dev

It fails with anything 3.10+ that I could find on conda-forge.

[luc-j-bourhis]

Not such luck: it fails much later. This is on MacOS 11.7.10 with 13.2.1 (clang 13.0.0), as this may now matter.

===== Running in build: rebuild rotarama
Traceback (most recent call last):
  File "/Users/luc/Developer/cctbx/modules/cctbx_project/boost_adaptbx/boost/python.py", line 22, in import_ext
    try: mod = __import__(name)
ImportError: dlopen(/Users/luc/Developer/cctbx/build/lib/scitbx_array_family_flex_ext.so, 2): Symbol not found: __ZN5boost6python9converter21object_manager_traitsINS0_5numpy7ndarrayEE10get_pytypeEv
  Referenced from: /Users/luc/Developer/cctbx/build/lib/scitbx_array_family_flex_ext.so
  Expected in: flat namespace
 in /Users/luc/Developer/cctbx/build/lib/scitbx_array_family_flex_ext.so

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/luc/Developer/cctbx/build/../modules/cctbx_project/mmtbx/command_line/rebuild_rotarama_cache.py", line 70, in <module>
    run()
  File "/Users/luc/Developer/cctbx/build/../modules/cctbx_project/mmtbx/command_line/rebuild_rotarama_cache.py", line 9, in run
    from mmtbx.rotamer import rotamer_eval
  File "/Users/luc/Developer/cctbx/modules/cctbx_project/mmtbx/rotamer/rotamer_eval.py", line 6, in <module>
    from mmtbx.rotamer.sidechain_angles import PropertyFile, SidechainAngles
  File "/Users/luc/Developer/cctbx/modules/cctbx_project/mmtbx/rotamer/sidechain_angles.py", line 2, in <module>
    import scitbx.math
  File "/Users/luc/Developer/cctbx/modules/cctbx_project/scitbx/math/__init__.py", line 7, in <module>
    from scitbx.math.ext import *
  File "/Users/luc/Developer/cctbx/modules/cctbx_project/scitbx/math/ext.py", line 2, in <module>
    import scitbx.array_family.flex # import dependency
  File "/Users/luc/Developer/cctbx/modules/cctbx_project/scitbx/array_family/flex.py", line 12, in <module>
    bp.import_ext("scitbx_array_family_flex_ext")
  File "/Users/luc/Developer/cctbx/modules/cctbx_project/boost_adaptbx/boost/python.py", line 31, in import_ext
    raise ImportError(
ImportError: __import__("scitbx_array_family_flex_ext"): dlopen(/Users/luc/Developer/cctbx/build/lib/scitbx_array_family_flex_ext.so, 2): Symbol not found: boost::python::converter::object_manager_traits<boost::python::numpy::ndarray>::get_pytype()
  Referenced from: /Users/luc/Developer/cctbx/build/lib/scitbx_array_family_flex_ext.so
  Expected in: flat namespace
 in /Users/luc/Developer/cctbx/build/lib/scitbx_array_family_flex_ext.so
  sys.path:
    /Users/luc/Developer/cctbx/modules/cctbx_project/mmtbx/command_line
    /Users/luc/Developer/cctbx/modules/cctbx_project
    /Users/luc/Developer/cctbx/modules
    /Users/luc/Developer/cctbx/modules/cctbx_project/boost_adaptbx
    /Users/luc/Developer/cctbx/build/lib
    /Users/luc/micromamba/envs/cctbx-dev/python.app/Contents/lib/python3.9/site-packages
    /Users/luc/micromamba/envs/cctbx-dev/python.app/Contents/lib/python39.zip
    /Users/luc/micromamba/envs/cctbx-dev/python.app/Contents/lib/python3.9
    /Users/luc/micromamba/envs/cctbx-dev/python.app/Contents/lib/python3.9/lib-dynload
Process failed with return code 1

[luc-j-bourhis]

Actually, I need numpy in my conda env but then I have to be careful about the version of Boost. Eventually, this conda spec

name: cctbx-dev
channels:
- https://conda.anaconda.org/conda-forge
- https://repo.anaconda.com/pkgs/main
dependencies:
- boost=1.73
- future
- numpy=1.25
- python.app==3=py39hca72f7f_0
- scons
- six

works with

python bootstrap.py --builder cctbxlite --use-conda ~/micromamba/envs/cctbx-dev --no-boost-src

[bkpoon]

The --use-conda flag assumes that the conda environment contains all the dependencies needed for building. There are general devenv files for cctbxlite and cctbx. I can generate cctbx and cctbxlite environment yaml files from them and add them. I do have to do an update on all the static environments anyway.

For Boost, you should be able to use the currently pinned version, 1.84, from conda-forge and that should work with numpy 2. The nightly conda packages are built with those versions and are tested nightly. If you need an earlier version of numpy, that conda package for Boost will also work with earlier versions.

[bkpoon]

I will also add the libboost conda packages to the environments. For CentOS 7, the compiler ABI is incompatible with the conda packages, so that is why we still had Boost as a directory in modules (and why the --no-boost-src flag was needed). But we should be able to move all platforms to use the Boost conda packages.

[luc-j-bourhis]

Thanks. Using your conda yaml specs, and adding boost = 1.84, I got the same compilation errors I got with the naive one I made at first. It has just downed on me that I missed --config_flags="--enable_cxx11". If I pass that to bootstrap.py, it compiles fine! I should have read more carefully the compiler error message, sorry.

[luc-j-bourhis]

It compiles but then it SIGDEV in space_group_symbols from cctbx_sgtbx_ext. So I guess, there is definitively a problem with a recent Boost…

[bkpoon]

What version of macOS and platform are you running? I can try building locally. We are still testing the Xcode compilers from Xcode 13, 14, and 15.

The minimum macOS version for the latest Boost packages is 10.13.

[luc-j-bourhis]

MacOS 11.7.10 with Xcode 13.2.1 (clang 13.0.0)

[bkpoon]

I made a new branch, newenv, with updated static environments for cctbxlite (macOS, Python 3.9). The yaml file is named test.yml. The build seems to be working on macOS 12 with Xcode 13.2.1 (test errors from Azure Pipelines are from a missing psutil dependency (https://dev.azure.com/cctbx/cctbx_project/_build/results?buildId=35129&view=results).

Can you send me the exact environment you're using that has the failure? You can get it by running conda list <conda environment name> --explicit --md5.

Another thing to do is to completely delete your build directory and try again. If you had built something before with the boost directory in modules. Those Boost libraries take precedence over the conda environment and the two versions are too different to be compatible.

[luc-j-bourhis]

Coming back to this. It might be that the problem only exists on MacOS 11 then. I will upgrade this computer in the near future, and I'll then be able to check your fix. Meanwhile, I restarted from scratch, and did just python bootstrap.py --builder cctbxlite --use-conda --nproc=9 build. Then bootstrap.py created a conda install for me, with Python 3.7. That's fine for what I wanted to do. I close this bug then. Thanks for your patience!