PERF: optimize is_scalar, is_iterator

[jbrockmendel]

While working on #30349 I noticed that is_scalar is pretty slow for non-scalar args, turns out we can get a 9-19x improvement for listlike cases, 5-10x improvement for Decimal/Period/DateOffset/Interval, with small improvements in nearly every other case while we're at it (the fractions.Fraction object is the only one where i found a small decrease in perf, within the margin of error)

xref #31291, assuming this is the desired behavior for is_iterator, this closes that.

Setup:

import pandas as pd                                                     
from pandas.core.dtypes.common import *                                 
import decimal, fractions

dec = decimal.Decimal(19.45678)
frac = fractions.Fraction(1)
i64 = np.int64(-1)
arr = np.arange(5)
ser = pd.Series(arr)
idx = pd.Index(arr)
interval = pd.Interval(0, 1)
per = pd.Period("2017")
offset = per - per
iterator = (x for x in [])

Non-Scalar Cases

In [6]: %timeit is_scalar([])
1.53 µs ± 36.4 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- master
79.7 ns ± 5.26 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)  # <-- PR

In [7]: %timeit is_scalar(arr)
1.58 µs ± 43.8 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- master
77.6 ns ± 5.02 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)  # <-- PR

In [8]: %timeit is_scalar(ser)
1.01 µs ± 51.7 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- master
92.3 ns ± 0.505 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)  # <-- PR

In [9]: %timeit is_scalar(idx)
876 ns ± 4.86 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- master
92.7 ns ± 1.47 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)  # <-- PR

In [10]: %timeit is_scalar(iterator)
2.08 µs ± 22.9 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  # <-- master
869 ns ± 20.9 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- PR

Non-Built-In Scalar Cases

In [10]: %timeit is_scalar(dec)
1.05 µs ± 9.75 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- master
79.5 ns ± 2.16 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)  # <-- PR

In [12]: %timeit is_scalar(interval)
748 ns ± 22.6 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- master
139 ns ± 5.38 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)  # <-- PR

In [13]: %timeit is_scalar(per)
706 ns ± 24.7 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- master
127 ns ± 14.2 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)  # <-- PR

In [14]: %timeit is_scalar(offset)
930 ns ± 40.6 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- master
211 ns ± 2.45 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- PR

Built-In Scalar Cases

In [3]: %timeit is_scalar(4)
60.1 ns ± 2.39 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)  # <-- master
54.9 ns ± 1.15 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)  # <-- PR

In [4]: %timeit is_scalar(4.0)
53 ns ± 4.94 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)  # <-- master
48.2 ns ± 0.613 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)  # <-- PR

In [5]: %timeit is_scalar(i64)
58.5 ns ± 1.39 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)  # <-- master
58.1 ns ± 5.88 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)  # <-- PR

In [11]: %timeit is_scalar(frac)
93.5 ns ± 2.05 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)  # <-- master
94.1 ns ± 6.4 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)  # <-- PR

is_iterator check

In [16] %timeit is_iterator(iterator)
283 ns ± 5.35 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- master
58.9 ns ± 2.31 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)  # <-- PR

In [17] %timeit is_iterator(arr)
228 ns ± 3.98 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- master
55 ns ± 3.53 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)  # <-- PR

In [17] %timeit is_iterator("foo")
231 ns ± 15.7 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- master
49.4 ns ± 5.08 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)  # <-- PR

[WillAyd]

lgtm

[WillAyd]

Could arguably just use PyIter_Check where needed now and avoid overhead of def call

[jbrockmendel]

so far no usage of this in the cython code, it is just replacing the version in core.dtypes.inference

[jreback]

sure

[jbrockmendel]

i was expecting you to be more excited about this!

[jreback]

lol, after going thru lots of PRs i type shortest things! this is good
😄