Faiss compatibility with NumPy 2 (ImportError: numpy.core.multiarray failed to import)

[pdarulewski]

Summary

I'm not really sure if this is something on my side or Faiss but from the logs it looks like Faiss needs to be recompiled with the newest NumPy release. AFAIK, there are a couple of C API changes: https://numpy.org/doc/stable/release/2.0.0-notes.html

Platform

OS: macOS 14.4.1 23E224 arm64

Faiss version: faiss-cpu==1.8.0

Installed from: pypi through poetry and pip

Faiss compilation options: none

Running on:

• CPU
• GPU

Interface:

• C++
• Python

Reproduction instructions

To reproduce:

1. install numpy, faiss-cpu, packaging (apparently is required by faiss but it's not automatically installed by pip 24.0)
2. open REPL and import faiss

∇ python3.11 -m venv .venv
∇ source .venv/bin/activate
∇ pip install numpy faiss-cpu packaging
Collecting numpy
  Using cached numpy-2.0.0-cp311-cp311-macosx_14_0_arm64.whl.metadata (60 kB)
Collecting faiss-cpu
  Using cached faiss_cpu-1.8.0-cp311-cp311-macosx_11_0_arm64.whl.metadata (3.6 kB)
Collecting packaging
  Using cached packaging-24.1-py3-none-any.whl.metadata (3.2 kB)
Using cached numpy-2.0.0-cp311-cp311-macosx_14_0_arm64.whl (5.2 MB)
Using cached faiss_cpu-1.8.0-cp311-cp311-macosx_11_0_arm64.whl (3.1 MB)
Using cached packaging-24.1-py3-none-any.whl (53 kB)
Installing collected packages: packaging, numpy, faiss-cpu
Successfully installed faiss-cpu-1.8.0 numpy-2.0.0 packaging-24.1
∇ python
Python 3.11.8 (main, Jun  7 2024, 13:22:20) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import faiss

A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.0 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

Traceback (most recent call last):  File "<stdin>", line 1, in <module>
  File "/Users/pd/utils/scratch/.venv/lib/python3.11/site-packages/faiss/__init__.py", line 16, in <module>
    from .loader import *
  File "/Users/pd/utils/scratch/.venv/lib/python3.11/site-packages/faiss/loader.py", line 98, in <module>
    from .swigfaiss import *
  File "/Users/pd/utils/scratch/.venv/lib/python3.11/site-packages/faiss/swigfaiss.py", line 10, in <module>
    from . import _swigfaiss
AttributeError: _ARRAY_API not found
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/pd/utils/scratch/.venv/lib/python3.11/site-packages/faiss/__init__.py", line 16, in <module>
    from .loader import *
  File "/Users/pd/utils/scratch/.venv/lib/python3.11/site-packages/faiss/loader.py", line 98, in <module>
    from .swigfaiss import *
  File "/Users/pd/utils/scratch/.venv/lib/python3.11/site-packages/faiss/swigfaiss.py", line 10, in <module>
    from . import _swigfaiss
ImportError: numpy.core.multiarray failed to import

[ydshieh]

Also see this in transformers CI while checking if it works with numpy v2

[mlomeli1]

hi @ydshieh, thank you for flagging. Just checking, if you installed faiss with pip, we don't support installation with pip. Could you try installing faiss with conda as per INSTALL.md and let us know if there are still issues?

[ydshieh]

Hi @mlomeli1 Our currently CI environment doesn't have conda in it, and so far we are able to install and use faiss-cpu along with numpy < 2.0, but not with numpy 2.0.0.

Is there a plan to support numpy 2.0.0 (and ideally with pip install)?

[pdarulewski]

@mlomeli1 thanks, I didn't know that, sorry about that. Unfortunately, if I use conda, it fetches numpy 1.26.4 and numpy-base 1.26.4:

∇ conda install -c pytorch faiss-cpu=1.8.0
[...]

  added / updated specs:
    - faiss-cpu=1.8.0


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    [...]
    numpy-1.26.4               |  py312h7f4fdc5_0          12 KB
    numpy-base-1.26.4          |  py312he047099_0         6.3 MB

Then I can't install numpy 2.0 because of version conflicts:

∇ conda install numpy=2.0.0
[...]
─ numpy 2.0.0**  is installable with the potential options
│  ├─ numpy [1.19.5|1.20.3|...|2.0.0], which can be installed (as previously explained);
│  ├─ numpy [1.21.3|1.21.4|...|2.0.0], which can be installed (as previously explained);
│  ├─ numpy [1.23.4|1.23.5|...|2.0.0], which can be installed (as previously explained);
│  ├─ numpy [2.0.0rc1|2.0.0rc2], which cannot be installed (as previously explained);
│  └─ numpy 2.0.0 conflicts with any installable versions previously reported;

@ydshieh There's an open issue already in faiss-wheels where faiss is published to pypi: kyamagu/faiss-wheels#104

[wencan]

@mlomeli1

I have always used pip + venv. This combination has always worked well.

[ydshieh]

@pdarulewski But #104 and #105 mean, as long as we need to use faiss, we can't use numpy v2. I don't feel strong, but maybe at some point it's nice to make them compatible.

[mengdilin]

Heads up, it looks like faiss + numpy2.0 using conda installation is also not behaving as expected from https://github.com/facebookresearch/faiss/actions/runs/10117066685/job/27981278918?pr=3696 I will take a look