Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix var list in rel node group #2315

Merged
merged 1 commit into from
Nov 1, 2023
Merged

Fix var list in rel node group #2315

merged 1 commit into from
Nov 1, 2023

Conversation

ray6080
Copy link
Contributor

@ray6080 ray6080 commented Oct 31, 2023

This is a temporary fix for COPY REL table with VAR_LIST properties.

Previously, we were storing VAR_LIST as ku_list entries plus overflow data in separate files. That had been changed for node tables already to a new layout, which consists of offsets and data . Details on the new layout can be found here.

Now as we are moving rel tables to node group based storage, we are also shifting the layout of VAR_LIST on rel tables to the new one. While the problem is that the new layout is not friendly to partially sorted insertions, as we are sorting tuples based on their src node offsets inside the node group.
Take an example of following tuples (dst is ignored):

src prop
10 [A,B,C]
100 [D]
10 [A,C,D]
100 [E,G,F]

The expected layout is:

offset: [3,6,7,10]
data:   [A,B,C,A,C,D,D,E,G,F]

Existing implementation of VAR_LIST column chunk will always append data to the end of the data column chunk, which would lead to

offset: [3,7,4,10]
data:   [A,B,C,D,A,C,D,E,G,F]

To solve this problem, this PR introduces an AuxVarListColumnChunk, which would keep lists as list_entry_t entires plus data column chunk, and after insertions (during which ColumnChunk::finalize() is called), every list inside it will be appended to a new VarListColumnChunk in a sorted way.

@ray6080 ray6080 merged commit 64171fe into node-group Nov 1, 2023
@ray6080 ray6080 deleted the node-group-var-list branch November 1, 2023 01:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants