Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use a lockfree data structure to store page states #3425

Merged
merged 2 commits into from
May 4, 2024

Conversation

benjaminwinger
Copy link
Collaborator

Note that this only supports safe reads without locks, adding new page states still needs locking, but it would be possible to support updates using atomics.

The ConcurrentVector is optimized in two ways:

  1. The initial size is allocated in one chunk, to minimize lookup time for existing elements
  2. New elements are added using a linked list of indices, which point to blocks of elements. With an appropriate size, the vast majority of situations should be able to use just one index block.

I currently have the sizes set up so that the access cost of the pageStates structure only increases with every 128GB of new file data. I also played around with increasing the page group size, but settled with using a large index size (256KB) instead as I wasn't sure about the possible side-effects of increasing the page group size.
The ConcurrentVector is also used for the frame group indices stored in the BMFileHandle as I noticed that there could be a data race in getFrameIdx if a new page group is added in another thread (very unlikely, as the frame group index vector is only appended to and not otherwise modified, as other threads would need to resize the vector, claim and then write to the freed memory between when a thread reads the frame group index vector's data pointer and accesses the data).

@benjaminwinger benjaminwinger force-pushed the page_state_lockfree branch 2 times, most recently from b79aeb3 to fa40928 Compare May 1, 2024 20:38
@benjaminwinger benjaminwinger force-pushed the page_state_lockfree branch 2 times, most recently from 57b2ad0 to ceae560 Compare May 3, 2024 18:54
@benjaminwinger
Copy link
Collaborator Author

It appears that statically linking kuzu.lib to the extensions on windows makes the ABI somewhat unstable.

@ray6080 ray6080 merged commit 5cd25bf into master May 4, 2024
26 checks passed
@ray6080 ray6080 deleted the page_state_lockfree branch May 4, 2024 00:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants