-
Notifications
You must be signed in to change notification settings - Fork 86
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Track variable sized memory manager allocations through the buffer manager #3564
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! Can we add some tests over this? We should add a new call function memory_usage()
, which returns currently used memory in bm.
For the tests, do we just want to make sure it's not increasing after queries? In the e2e tests, is there a way of storing and comparing variables? Otherwise it seems like it would be easier to write a C++ test that could check both during and after the memory is allocated, but that wouldn't need a call function (we could have both I guess). Page caching will also be an issue with that, as unless we also add a way to drop those caches we won't be able to guarantee that the memory usage won't go up or down after a query if new pages are cached, or if old pages are dropped to make space. |
Yeah, I think we just want to make sure that the mem usage is not going up after re-run same small queries. The case could be we record the There isn't a way of storing variables now in testing framework, might be easier to do through CPP instead of Cypher. |
7df52ad
to
ca82cf1
Compare
I've added a simple test. |
Benchmark ResultMaster commit hash:
|
To follow up the missing part of #3243, this adds BufferManager tracking for variable-sized memory allocations made through the MemoryManager.
I also noticed that the buffer manager wasn't freeing the memory related to evictions if it fails to allocate the total amount of memory requested when claiming a frame. I don't know how much of our code at the moment can safely recover from failing to claim memory and continue allocations though, so it may not make a difference at the moment. I had noticed subsequent failures which seemed to be related to data not being successfully written in the multi copy test that was failing due to not having enough memory in #3557 (e.g. accessing out of bounds in a hash index disk array, which I think would have been in bounds if the data had been updated correctly), so at the moment it may not always be possible to continue using the database after such a failure.