BUG: isin has unexpected behaviour when using a dataframe with a large amount of rows (1.000.001)

[Hanspagh]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

import pandas as pd
import numpy as np
nans = np.repeat(np.nan, 1_000_001)
big = pd.DataFrame({"test":nans})
assert all(big["test"].isin([np.nan])) == True
 
nans = np.repeat(np.nan, 1_000_000)
small = pd.DataFrame({"test":nans})
assert all(small["test"].isin([np.nan])) == True
assert all(small.dtypes == big.dtypes)

Problem description

The above code fails on the first assert, but not on the second.

Expected Output

The excepted behavior would be that both asserts either fail or succeeds

Output of pd.show_versions()

INSTALLED VERSIONS

commit : f2ca0a2
python : 3.7.7.final.0
python-bits : 64
OS : Darwin
OS-release : 19.3.0
Version : Darwin Kernel Version 19.3.0: Thu Jan 9 20:58:23 PST 2020; root:xnu-6153.81.5~1/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.1.1
numpy : 1.19.1
pytz : 2020.1
dateutil : 2.8.1
pip : 19.2.3
setuptools : 41.2.0
Cython : 0.29.17
pytest : 5.4.3
hypothesis : 4.57.1
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.5.2
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.16.1
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.3.0
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 0.15.1
pytables : None
pyxlsb : None
s3fs : None
scipy : 1.5.2
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : 0.48.0

[jreback]

i think this is patched on master - pls check

[Hanspagh]

It does not seem to be patched on master.
Running with the lastest master produces the same bug

------------------
commit           : 361166f86da1a665a1b5808e0935c28f8cf56d34
python           : 3.7.7.final.0
python-bits      : 64
OS               : Darwin
OS-release       : 19.3.0
Version          : Darwin Kernel Version 19.3.0: Thu Jan  9 20:58:23 PST 2020; root:xnu-6153.81.5~1/RELEASE_X86_64
machine          : x86_64
processor        : i386
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 1.2.0.dev0+233.g361166f86
numpy            : 1.19.0rc2
pytz             : 2020.1
dateutil         : 2.8.1
pip              : 20.1.1
setuptools       : 41.2.0
Cython           : 0.29.21
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 2.10.3
IPython          : None
pandas_datareader: None
bs4              : None
bottleneck       : None
fsspec           : None
fastparquet      : None
gcsfs            : None
matplotlib       : None
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : 1.0.1
pytables         : None
pyxlsb           : None
s3fs             : None
scipy            : None
sqlalchemy       : None
tables           : None
tabulate         : 0.8.7
xarray           : None
xlrd             : None
xlwt             : None
numba            : None

[jreback]

pls look for a duplicate bug report

[Hanspagh]

Simular problem with series #25395
might be loosely related to #35565 and #34125

[Hanspagh]

Seems to boil down to the use of np.in1d for more than 1mill rows, would it be better to disallow NaN in isin, since

NaN values never compare equal. That is, the test NaN==NaN is always False by definition of NaN.

and then ask the user to use isna instead?

[jreback]

hmm we do properly handle man actually for smaller sizes

so would need to filter null out and then add back if it exists (and isin) as we want the perf of np.in1d for larger sizes

[TomAugspurger]

I think fixing #25395 will fix this as well, so closing as a duplicate of that.