Use siphash for persistent hashing #242
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I don't know how much of a bottleneck the hash computation is in mirgecom compiles, but at the same time, I can't think of a good reason to not do this. Our use case very much falls into the "many small messages" category, so this graph is pretty compelling:
(from https://cr.yp.to/siphash/siphash-20120918.pdf)
A few things need to happen in order to make this go:
siphash24 would need to be packaged for conda-forge (presumably straightforward)(This can wait until next release and we package this for conda-forge.)hexdigest
should returnstr
, notbytes
dnicolodi/python-siphash24#6 (though we could hack around that)hash_constructor
in call tounordered_hash
loopy#849cc @matthiasdiener