BUG: CategoricalIndex.get_indexer does not detect duplicates

[batterseapower]

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of pandas (checked on 1.1.5, the latest Anaconda version)
• (optional) I have confirmed this bug exists on the master branch of pandas.

---

pd.CategoricalIndex(['A', 'B', 'C', 'A', 'B']).get_indexer(pd.CategoricalIndex(['B', 'C', 'D', 'E']))
# returns array([ 1,  4,  2, -1, -1], dtype=int64)
pd.Index(['A', 'B', 'C', 'A', 'B']).get_indexer(pd.Index(['B', 'C', 'D', 'E']))
# InvalidIndexError: Reindexing only valid with uniquely valued Index objects

Problem description

CategoricalIndex.get_indexer should fail if the index contains duplicates, just like the regular index does. Instead, it returns a nonsense array that does not even have the same length as the target.

Note this problem also affects the case where dtypes are equal or the target is not categorical i.e.:

# 1
dt = pd.CategoricalDtype(categories=['A', 'B', 'C', 'D', 'E'])
pd.CategoricalIndex(['A', 'B', 'C', 'A', 'B'], dtype=dt).get_indexer(pd.CategoricalIndex(['B', 'C', 'D', 'E'], dtype=dt))

# 2
pd.CategoricalIndex(['A', 'B', 'C', 'A', 'B']).get_indexer(pd.Index(['B', 'C', 'D', 'E']))

I fixed a related bug in Sep 2019 as #28257 but that fix was merged in v1.0.0 so this must be a seperate issue or a regression. Could also be related to #25459.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : b5958ee
python : 3.8.5.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.18362
machine : AMD64
processor : Intel64 Family 6 Model 85 Stepping 7, GenuineIntel
byteorder : little
LC_ALL : None
LANG : en_GB.UTF-8
LOCALE : English_United Kingdom.1252

pandas : 1.1.5
numpy : 1.19.2
pytz : 2020.4
dateutil : 2.8.1
pip : 20.3.3
setuptools : 51.0.0.post20201207
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[jreback]

try on 1.2.0rc0 / master

lots of things have been fixed

[batterseapower]

After some searching I found the instructions for installing a pre-release version on https://github.com/pandas-dev/pandas/releases.

The problem still exists in 1.2.0rc0.

INSTALLED VERSIONS

commit : 7688d3c
python : 3.8.5.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.18362
machine : AMD64
processor : Intel64 Family 6 Model 85 Stepping 7, GenuineIntel
byteorder : little
LC_ALL : None
LANG : en_GB.UTF-8
LOCALE : English_United Kingdom.1252

pandas : 1.2.0rc0
numpy : 1.19.2
pytz : 2020.4
dateutil : 2.8.1
pip : 20.3.3
setuptools : 51.0.0.post20201207
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : 1.3.2
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : 3.0.5
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : 1.5.2
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : 0.51.2

[simonjayhawkins]

but is fixed on master

[simonjayhawkins]

but is fixed on master

fixed in [5a7514c] REF: de-duplicate get_indexer methods (#38372)

and tests in pandas/tests/indexes/categorical/test_indexing.py updated.

[MarcoGorelli]

As far as I can tell, based on the above, this can be closed, but please do ping me if my assessment is wrong