Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework var list finalize #2392

Merged
merged 1 commit into from
Nov 11, 2023
Merged

Rework var list finalize #2392

merged 1 commit into from
Nov 11, 2023

Conversation

ray6080
Copy link
Contributor

@ray6080 ray6080 commented Nov 11, 2023

VarListColumnChunk::write is expected to write a var list value to a given position inside the column chunk, and there is no enforcement of the order of given positions. For example, write can take [10,20] to pos 5, [1,2,8] to pos 1, etc.
However, the layout of var list column requires offsets to be sorted based on positions in the column chunk.

This PR introduces an extra indices column chunk to handle the ordering difference. During write, an extra index is recorded into indices to track where the var list value is stored. And finalize is called after writes are done, which re-append var list values according to their position in the column chunk.

Copy link

codecov bot commented Nov 11, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (24934c6) 91.18% compared to head (f04406a) 91.20%.
Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2392      +/-   ##
==========================================
+ Coverage   91.18%   91.20%   +0.01%     
==========================================
  Files        1012     1012              
  Lines       35933    35933              
==========================================
+ Hits        32767    32771       +4     
+ Misses       3166     3162       -4     
Files Coverage Δ
src/include/storage/store/column_chunk.h 97.61% <100.00%> (ø)
src/include/storage/store/node_group.h 100.00% <100.00%> (ø)
src/include/storage/store/struct_column_chunk.h 100.00% <ø> (ø)
src/include/storage/store/var_list_column_chunk.h 100.00% <100.00%> (+9.09%) ⬆️
src/processor/operator/persistent/copy_rel.cpp 96.42% <100.00%> (ø)
src/storage/store/column_chunk.cpp 95.91% <100.00%> (-0.05%) ⬇️
src/storage/store/node_group.cpp 100.00% <100.00%> (ø)
src/storage/store/struct_column_chunk.cpp 73.33% <100.00%> (+2.60%) ⬆️
src/storage/store/var_list_column_chunk.cpp 96.62% <100.00%> (ø)

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

src/include/storage/store/struct_column_chunk.h Outdated Show resolved Hide resolved
src/include/storage/store/var_list_column_chunk.h Outdated Show resolved Hide resolved
src/storage/store/var_list_column_chunk.cpp Outdated Show resolved Hide resolved
@ray6080 ray6080 merged commit 21c03f6 into master Nov 11, 2023
12 checks passed
@ray6080 ray6080 deleted the var-list-finalize branch November 11, 2023 06:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants