Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Paired Block Bloom Filter Algorith - BPK Adjustment to avoid performance degradation with bpk<=10 #161

Closed
udi-speedb opened this issue Sep 15, 2022 · 6 comments · Fixed by #163
Assignees
Labels
enhancement New feature or request performance
Milestone

Comments

@udi-speedb
Copy link
Contributor

udi-speedb commented Sep 15, 2022

During @noamhaham's performance runs, he discovered that there is a performance degradation with the paired filter with bpk=10.
suggests to adjust the user's bpk (round up) to end up in an effective bpk that is larger that the user's. This adjustment implies more memory consumption for the filters (relative to not rounding up). The more keys a filter has, the smaller the adjustment / memory-consumption-increase and vice versa =>
Noam expects this to be felt more in Level 0 and less as the level progresses. However, since most of the data is in the lower levels, he expects the total consumption increase to be negligible.

@udi-speedb udi-speedb self-assigned this Sep 15, 2022
@udi-speedb udi-speedb added the enhancement New feature or request label Sep 15, 2022
@udi-speedb
Copy link
Contributor Author

Pull request: #163

@udi-speedb udi-speedb pinned this issue Sep 15, 2022
@isaac-io isaac-io linked a pull request Sep 19, 2022 that will close this issue
@Guyme Guyme assigned erez-speedb and noamhaham and unassigned erez-speedb Sep 21, 2022
@isaac-io isaac-io added this to the v2.2.0 milestone Oct 26, 2022
@isaac-io isaac-io changed the title Paired Block Bloom Filter Algorith - BPK Adjustment to avoid performance degradation with bpk<=8 Paired Block Bloom Filter Algorith - BPK Adjustment to avoid performance degradation with bpk<=10 Oct 26, 2022
@Yuval-Ariel Yuval-Ariel unpinned this issue Nov 27, 2022
@erez-speedb erez-speedb self-assigned this Dec 5, 2022
@erez-speedb
Copy link

Performance needs to cover a few tests.
Checking with 20 BPK and compare to previous version, make sure no degradation.
Test on 8 BPK and compare to native bloom.

@erez-speedb
Copy link

Test on the 20 BPS show some degradation,

Image

The second test was OK

Image

@erez-speedb
Copy link

The 8 BPS test

Image

@erez-speedb
Copy link

Pass performance tests

@udi-speedb
Copy link
Contributor Author

@erez says he has checked a branch named pared_bloom_filter-improved
I will take a look at it now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request performance
Projects
Status: ✅ Shipped
Development

Successfully merging a pull request may close this issue.

5 participants