PERF: Introducing hash tables for complex64 and complex128 #38179

realead · 2020-11-30T09:39:23Z

tests added / passed
passes black pandas
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

Next step for #33287

The complex case is somewhat less straight forward, because Cython defines complex128/complex64 as:

ctypedef float complex  complex64_t
ctypedef double complex complex128_t

where double complex actually means a Cython-datatype defined here https://github.com/cython/cython/blob/master/Cython/Utility/Complex.c#L56

Thus a conversion between numpy's complex and Cython's complex must happens somewhere.

realead · 2020-11-30T09:48:58Z

FYI @jbrockmendel

Some polishing is still needed and probably additional testing, but this is how it could look like.

Maybe you know a better solution for the whole numpy's <->cython's complex-business.

There are also some details, which maybe need some tinkering:

Right now, nan+1j and 1+nan*j are considered not-a-number. However, they aren't considered equivalent. Not sure that doesn't make any problems.
inf +nan*j is considered not-a-number, maybe it makes more sense to see it as infinity and not not-a-number

pandas/_libs/hashtable_class_helper.pxi.in

jbrockmendel · 2020-12-01T02:48:51Z

pandas/_libs/hashtable_class_helper.pxi.in

@@ -606,14 +681,15 @@ cdef class {{name}}HashTable(HashTable):
                                 ignore_na=True, return_inverse=True)
        return labels

+    {{if dtype == 'int64'}}


why adding this only for int64?

This function is used just in int64-version:

pandas/pandas/core/sorting.py

Lines 598 to 603 in aad85ad

table = hashtable.Int64HashTable(size_hint)

group_index = ensure_int64(group_index)

# note, group labels come out ascending (ie, 1,2,3 etc)

comp_ids, obs_group_ids = table.get_labels_groupby(group_index)

The logic in this function:

pandas/pandas/_libs/hashtable_class_helper.pxi.in

Lines 628 to 633 in 1fc5efd

# specific for groupby

{{if dtype != 'uint64'}}

if val < 0:

labels[i] = -1

continue

{{endif}}

IIUC, negative numbers are special and should be skipped, but this is only a convention for the meaning of values group_index. For other types there is no such convention. Instead of inventing something, I've decided to delete unused functions/versions.

pandas/_libs/hashtable_class_helper.pxi.in

pandas/_libs/hashtable_func_helper.pxi.in

pandas/tests/libs/test_hashtable.py

realead · 2020-12-01T22:26:38Z

Failures seem to be unrelated to my changes...

jbrockmendel · 2020-12-02T00:04:12Z

Failures seem to be unrelated to my changes...

Yes. The npdev failure I think should be fixed by #38209 and the other one just annoyingly happens sometimes because hypothesis is slow. Don't worry about either.

jreback

looks good a few questions.

jreback · 2020-12-02T02:54:59Z

pandas/tests/libs/test_hashtable.py

@@ -80,6 +82,8 @@ def test_map(self, table_type, dtype):
            table = table_type()
            keys = np.arange(N).astype(dtype)
            vals = np.arange(N).astype(np.int64) + N
+            keys.flags.writeable = False


can you parameterize all of these on writeable True/False

Done: 073222e

jreback · 2020-12-02T02:56:25Z

pandas/_libs/hashtable_class_helper.pxi.in

+
+{{for c_type in float_types}}
+
+cdef bint is_nan_{{c_type}}({{c_type}} val) nogil:


we don't define this for int types (as a no-op)? would it make the logic simpler if we did this (then we can always call a parameterized is_nan ?

Actually, we already do. I've reshuffled the code, so it becomes more obvious.

jreback · 2020-12-29T17:25:09Z

can you merge master and will look again

…ill not be dtype_t

…2, so making a workaround

…nly for int64. What should happen for other types (encoding nans as negative values is not possible with all types) depends on how it is used (but it is not used right now)

…d for which c_types

jreback · 2020-12-29T23:30:25Z

ok looks good. this complex 64/128 are available on all platforms?

realead · 2020-12-30T17:31:45Z

ok looks good. this complex 64/128 are available on all platforms?

@jreback It should work on platforms which have 64bit- and 32bit floating numbers. As we don't really use complex arithmetic it is not important whether C-compiler supports complex-numbers or not.

jreback · 2020-12-30T18:31:41Z

thanks @realead

…v#38179)

realead commented Nov 30, 2020

View reviewed changes

pandas/_libs/hashtable_class_helper.pxi.in Outdated Show resolved Hide resolved

jbrockmendel reviewed Dec 1, 2020

View reviewed changes

pandas/_libs/hashtable_class_helper.pxi.in Show resolved Hide resolved

jbrockmendel reviewed Dec 1, 2020

View reviewed changes

pandas/_libs/hashtable_func_helper.pxi.in Outdated Show resolved Hide resolved

jbrockmendel reviewed Dec 1, 2020

View reviewed changes

pandas/_libs/hashtable_func_helper.pxi.in Show resolved Hide resolved

jbrockmendel reviewed Dec 1, 2020

View reviewed changes

pandas/tests/libs/test_hashtable.py Outdated Show resolved Hide resolved

realead changed the title ~~WIP: PERF: Introducing hash tables for complex64 and complex128~~ PERF: Introducing hash tables for complex64 and complex128 Dec 1, 2020

jreback requested changes Dec 2, 2020

View reviewed changes

jreback added Complex Complex Numbers Performance Memory or execution speed performance labels Dec 2, 2020

realead force-pushed the complex_for_khash2 branch from e73bb2c to 0cce398 Compare December 2, 2020 21:26

jbrockmendel mentioned this pull request Dec 8, 2020

Complex Dtype Support for Hashmap Algos #36482

Merged

4 tasks

realead added 14 commits December 29, 2020 21:38

remove unnecessary setting of keys kh_put_xxx, already does it!

0b70965

adding explicit c_type as preparation for complex, for which c_type w…

06f7a07

…ill not be dtype_t

introducing complex hash tables

6fe568e

ensure const-correctness of input is tested

fbbf544

const memoryviews of struct dtypes will work only with Cython>=0.29.2…

1fe2d55

…2, so making a workaround

making count_values and mode accept constand input as well

57e2a72

activating tests for complex hashtables

259deff

using to_c_type, rather than a switch

22ee6e7

getting rid of complex_group/float_group

5bcd106

fixing style issues

432f839

removing commented out code

a4903b0

get_labels_groupby is only used with Int64HashTable, thus define it o…

bb1f3d5

…nly for int64. What should happen for other types (encoding nans as negative values is not possible with all types) depends on how it is used (but it is not used right now)

fixing test case for 32bit build

28c2f2d

minor style issues

50d7136

realead added 2 commits December 29, 2020 21:38

parameterize tests on writeable True/False

4efdc67

reshuffle code, so it becomes more obvious which functions are define…

9930a55

…d for which c_types

realead force-pushed the complex_for_khash2 branch from 0cce398 to 9930a55 Compare December 29, 2020 20:42

jreback added this to the 1.3 milestone Dec 29, 2020

jreback approved these changes Dec 29, 2020

View reviewed changes

jreback merged commit f1d4026 into pandas-dev:master Dec 30, 2020

realead deleted the complex_for_khash2 branch December 31, 2020 08:01

luckyvs1 pushed a commit to luckyvs1/pandas that referenced this pull request Jan 20, 2021

PERF: Introducing hash tables for complex64 and complex128 (pandas-de…

9f346e7

…v#38179)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PERF: Introducing hash tables for complex64 and complex128 #38179

PERF: Introducing hash tables for complex64 and complex128 #38179

realead commented Nov 30, 2020 •

edited

Loading

realead commented Nov 30, 2020

jbrockmendel Dec 1, 2020

realead Dec 1, 2020

realead commented Dec 1, 2020 •

edited

Loading

jbrockmendel commented Dec 2, 2020

jreback left a comment

jreback Dec 2, 2020

realead Dec 2, 2020

jreback Dec 2, 2020

realead Dec 2, 2020

jreback commented Dec 29, 2020

jreback commented Dec 29, 2020

realead commented Dec 30, 2020

jreback commented Dec 30, 2020

	table = hashtable.Int64HashTable(size_hint)

	group_index = ensure_int64(group_index)

	# note, group labels come out ascending (ie, 1,2,3 etc)
	comp_ids, obs_group_ids = table.get_labels_groupby(group_index)

	# specific for groupby
	{{if dtype != 'uint64'}}
	if val < 0:
	labels[i] = -1
	continue
	{{endif}}


		{{for c_type in float_types}}

		cdef bint is_nan_{{c_type}}({{c_type}} val) nogil:

PERF: Introducing hash tables for complex64 and complex128 #38179

PERF: Introducing hash tables for complex64 and complex128 #38179

Conversation

realead commented Nov 30, 2020 • edited Loading

realead commented Nov 30, 2020

jbrockmendel Dec 1, 2020

Choose a reason for hiding this comment

realead Dec 1, 2020

Choose a reason for hiding this comment

realead commented Dec 1, 2020 • edited Loading

jbrockmendel commented Dec 2, 2020

jreback left a comment

Choose a reason for hiding this comment

jreback Dec 2, 2020

Choose a reason for hiding this comment

realead Dec 2, 2020

Choose a reason for hiding this comment

jreback Dec 2, 2020

Choose a reason for hiding this comment

realead Dec 2, 2020

Choose a reason for hiding this comment

jreback commented Dec 29, 2020

jreback commented Dec 29, 2020

realead commented Dec 30, 2020

jreback commented Dec 30, 2020

realead commented Nov 30, 2020 •

edited

Loading

realead commented Dec 1, 2020 •

edited

Loading