make failure on mac - ld: Undefined symbols:

[naveentatikonda]

Summary

make is failing on MacOS "with Undefined symbols error" when building from source on the latest code from main branch with Python. Able to build it successfully with C++

Error log

[  7%] Linking CXX shared library libfaiss_python_callbacks.dylib
make[3]: Leaving directory '/Users/navtat/Documents/git-repos/faiss/build'
[ 96%] Built target faiss
ld: Undefined symbols:
  _PyBytes_AsString, referenced from:
      PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o
  _PyBytes_FromStringAndSize, referenced from:
      PyCallbackIOWriter::operator()(void const*, unsigned long, unsigned long) in python_callbacks.cpp.o
  _PyBytes_Size, referenced from:
      PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o
  _PyGILState_Ensure, referenced from:
      PyCallbackIOWriter::PyCallbackIOWriter(_object*, unsigned long) in python_callbacks.cpp.o
      PyCallbackIOWriter::operator()(void const*, unsigned long, unsigned long) in python_callbacks.cpp.o
      PyCallbackIOWriter::~PyCallbackIOWriter() in python_callbacks.cpp.o
      PyCallbackIOReader::PyCallbackIOReader(_object*, unsigned long) in python_callbacks.cpp.o
      PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o
      PyCallbackIOReader::~PyCallbackIOReader() in python_callbacks.cpp.o
      PyCallbackIDSelector::PyCallbackIDSelector(_object*) in python_callbacks.cpp.o
      ...
  _PyGILState_Release, referenced from:
      PyCallbackIOWriter::PyCallbackIOWriter(_object*, unsigned long) in python_callbacks.cpp.o
      PyCallbackIOWriter::PyCallbackIOWriter(_object*, unsigned long) in python_callbacks.cpp.o
      PyCallbackIOWriter::operator()(void const*, unsigned long, unsigned long) in python_callbacks.cpp.o
      PyCallbackIOWriter::operator()(void const*, unsigned long, unsigned long) in python_callbacks.cpp.o
      PyCallbackIOWriter::~PyCallbackIOWriter() in python_callbacks.cpp.o
      PyCallbackIOWriter::~PyCallbackIOWriter() in python_callbacks.cpp.o
      PyCallbackIOReader::PyCallbackIOReader(_object*, unsigned long) in python_callbacks.cpp.o
      PyCallbackIOReader::PyCallbackIOReader(_object*, unsigned long) in python_callbacks.cpp.o
      ...
  _PyObject_CallFunction, referenced from:
      PyCallbackIOWriter::operator()(void const*, unsigned long, unsigned long) in python_callbacks.cpp.o
      PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o
      PyCallbackIDSelector::is_member(long long) const in python_callbacks.cpp.o
  _PyObject_IsTrue, referenced from:
      PyCallbackIDSelector::is_member(long long) const in python_callbacks.cpp.o
  __Py_Dealloc, referenced from:
      PyCallbackIOWriter::operator()(void const*, unsigned long, unsigned long) in python_callbacks.cpp.o
      PyCallbackIOWriter::~PyCallbackIOWriter() in python_callbacks.cpp.o
      PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o
      PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o
      PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o
      PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o
      PyCallbackIOReader::~PyCallbackIOReader() in python_callbacks.cpp.o
      ...
  faiss::FaissException::FaissException(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, char const*, char const*, int), referenced from:
      PyCallbackIOWriter::operator()(void const*, unsigned long, unsigned long) in python_callbacks.cpp.o
      PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o
      PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o
      PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o
      PyCallbackIDSelector::is_member(long long) const in python_callbacks.cpp.o
      PyCallbackIDSelector::is_member(long long) const in python_callbacks.cpp.o
  faiss::IOReader::filedescriptor(), referenced from:
      vtable for PyCallbackIOReader in python_callbacks.cpp.o
  faiss::IOWriter::filedescriptor(), referenced from:
      vtable for PyCallbackIOWriter in python_callbacks.cpp.o
  typeinfo for faiss::FaissException, referenced from:
      PyCallbackIOWriter::operator()(void const*, unsigned long, unsigned long) in python_callbacks.cpp.o
      PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o
      PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o
      PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o
      PyCallbackIDSelector::is_member(long long) const in python_callbacks.cpp.o
      PyCallbackIDSelector::is_member(long long) const in python_callbacks.cpp.o
  typeinfo for faiss::IOReader, referenced from:
      typeinfo for PyCallbackIOReader in python_callbacks.cpp.o
  typeinfo for faiss::IOWriter, referenced from:
      typeinfo for PyCallbackIOWriter in python_callbacks.cpp.o
  vtable for faiss::FaissException, referenced from:
      faiss::FaissException::~FaissException() in python_callbacks.cpp.o
  vtable for faiss::IOReader, referenced from:
      PyCallbackIOReader::PyCallbackIOReader(_object*, unsigned long) in python_callbacks.cpp.o
      PyCallbackIOReader::~PyCallbackIOReader() in python_callbacks.cpp.o
  vtable for faiss::IOWriter, referenced from:
      PyCallbackIOWriter::PyCallbackIOWriter(_object*, unsigned long) in python_callbacks.cpp.o
      PyCallbackIOWriter::~PyCallbackIOWriter() in python_callbacks.cpp.o
      PyCallbackIOWriter::~PyCallbackIOWriter() in python_callbacks.cpp.o
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[3]: *** [faiss/python/CMakeFiles/faiss_python_callbacks.dir/build.make:97: faiss/python/libfaiss_python_callbacks.dylib] Error 1
make[3]: Leaving directory '/Users/navtat/Documents/git-repos/faiss/build'
make[2]: *** [CMakeFiles/Makefile2:1260: faiss/python/CMakeFiles/faiss_python_callbacks.dir/all] Error 2
make[2]: Leaving directory '/Users/navtat/Documents/git-repos/faiss/build'
make[1]: *** [CMakeFiles/Makefile2:1107: faiss/python/CMakeFiles/swigfaiss.dir/rule] Error 2
make[1]: Leaving directory '/Users/navtat/Documents/git-repos/faiss/build'
make: *** [Makefile:582: swigfaiss] Error 2

Platform

OS: MacOS 13.6.7 (Intel)
SWIG Version: 4.2.1

Running on:

• CPU
• GPU

Interface:

• C++
• Python

Reproduction instructions

cmake -B build \
              -DBUILD_SHARED_LIBS=ON \
              -DFAISS_ENABLE_GPU=OFF \
              -DFAISS_ENABLE_C_API=ON \
              -DFAISS_ENABLE_PYTHON=ON \
              .

make -C build -j swigfaiss