faiss + transformers = SIGSEGV

[ikamensh]

Summary

harmless - looking combination of imports causes SIGSEGV.

Platform

OS: macOS Version 14.5 (23F79)
Hardware: Apple M3 Pro

Faiss version: pip freeze -> faiss==1.8.0
Installed from: anaconda, cpu version

Running on:

• CPU
• GPU

Interface:

• C++
• Python

Reproduction instructions

have foo.py:

import faiss
from transformers import AutoModel

# Load pre-trained model and tokenizer
model_name = "sentence-transformers/all-MiniLM-L6-v2"
model = AutoModel.from_pretrained(model_name)

Output:

/opt/homebrew/Caskroom/miniconda/base/envs/argus312/bin/python /Users/kai7rng/PycharmProjects/argus/foo.py 

Process finished with exit code 139 (interrupted by signal 11:SIGSEGV)

All versions:
~/P/argus (main↑1|✚2)$ pip freeze (argus312) 21:13:29
annotated-types @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_1fa2djihwb/croot/annotated-types_1709542925772/work
anyio==4.4.0
-e git+https://github.com/ikamensh/argus.git@58ac98b29dee07445c50e3c1e7d840e3648b83d9#egg=argus
certifi @ file:///private/var/folders/k1/30mswbxs7r1g6zwn8y4fyt500000gp/T/abs_70oty9s9jh/croot/certifi_1720453497032/work/certifi
charset-normalizer==3.3.2
click==8.1.7
distro @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_ddkyz0575y/croot/distro_1714488254309/work
dnspython==2.6.1
email_validator==2.2.0
faiss==1.8.0
fastapi==0.111.0
fastapi-cli==0.0.4
filelock==3.15.4
fsspec==2024.6.1
h11 @ file:///private/var/folders/k1/30mswbxs7r1g6zwn8y4fyt500000gp/T/abs_110bmw2coo/croot/h11_1706652289620/work
httpcore==1.0.5
httptools==0.6.1
httpx @ file:///private/var/folders/k1/30mswbxs7r1g6zwn8y4fyt500000gp/T/abs_727e6zfsxn/croot/httpx_1706887102687/work
huggingface-hub==0.23.4
idna @ file:///private/var/folders/k1/30mswbxs7r1g6zwn8y4fyt500000gp/T/abs_a12xpo84t2/croot/idna_1714398852854/work
iniconfig==2.0.0
Jinja2==3.1.4
joblib==1.4.2
markdown-it-py==3.0.0
MarkupSafe==2.1.5
mdurl==0.1.2
mpmath==1.3.0
mypy==1.10.1
mypy-extensions==1.0.0
networkx==3.3
numpy @ file:///private/var/folders/k1/30mswbxs7r1g6zwn8y4fyt500000gp/T/abs_a51i_mbs7m/croot/numpy_and_numpy_base_1708638620867/work/dist/numpy-1.26.4-cp312-cp312-macosx_11_0_arm64.whl#sha256=37afb6b734a197702d848df93bd67c10b52f6467d56e518950d84b6b1c949d27
openai @ file:///private/var/folders/k1/30mswbxs7r1g6zwn8y4fyt500000gp/T/abs_2745p20hqe/croot/openai_1714743709347/work
orjson==3.10.6
packaging @ file:///private/var/folders/k1/30mswbxs7r1g6zwn8y4fyt500000gp/T/abs_81ri4yfpjw/croot/packaging_1720101866878/work
pluggy==1.5.0
pydantic @ file:///private/var/folders/k1/30mswbxs7r1g6zwn8y4fyt500000gp/T/abs_0ai8cvgm2c/croot/pydantic_1709577986211/work
pydantic_core==2.20.1
Pygments==2.18.0
pytest==8.2.2
python-dotenv==1.0.1
python-multipart==0.0.9
PyYAML==6.0.1
regex==2024.5.15
requests==2.32.3
rich==13.7.1
ruff==0.5.1
safetensors==0.4.3
scikit-learn==1.5.1
scipy==1.14.0
setuptools==69.5.1
shellingham==1.5.4
sniffio==1.3.1
starlette==0.37.2
sympy==1.13.0
threadpoolctl==3.5.0
tokenizers==0.19.1
torch==2.3.1
tqdm @ file:///private/var/folders/k1/30mswbxs7r1g6zwn8y4fyt500000gp/T/abs_f76_dxtcsh/croot/tqdm_1716395948224/work
transformers==4.42.3
typer==0.12.3
typing_extensions @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_93dg13ilv4/croot/typing_extensions_1715268840722/work
ujson==5.10.0
urllib3==2.2.2
uvicorn==0.30.1
uvloop==0.19.0
watchfiles==0.22.0
websockets==12.0
wheel==0.43.0
~/P/argus (main↑1|✚2)$ python -V (argus312) 21:13:41
Python 3.12.4

[asadoughi]

I don't have an M3 Pro, but on an M1 Pro this worked fine (exit code 0) with the source code you provided:

conda install -c pytorch faiss && conda install -c conda-forge transformers
conda install -c pytorch pytorch::pytorch
python3 foo.py

[scriptator]

I ran into the same issue I ended up with exactly the same minimal reproducible example when trying to pin it down. In the end I found out that the issue is resolved when upgrading transformers (4.42.4 -> 4.43.3). Maybe you could try this as well @ikamensh ?

[github-actions]

This issue is stale because it has been open for 7 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 7 days since being marked as stale.