Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework arrow array copy in node table copier #1469

Merged
merged 1 commit into from
Apr 17, 2023
Merged

Rework arrow array copy in node table copier #1469

merged 1 commit into from
Apr 17, 2023

Conversation

ray6080
Copy link
Contributor

@ray6080 ray6080 commented Apr 16, 2023

Get rid of the toString() conversion in copying from arrow to column chunk in node copier.
This can boost the node copy performance a lot.
On uw5, the copy of ldbc-100 comment csv file can be reduced from 1031s to 286s.

Copy link
Contributor

@andyfengHKU andyfengHKU left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't forget to update description

@codecov
Copy link

codecov bot commented Apr 16, 2023

Codecov Report

Patch coverage: 92.12% and project coverage change: -0.03 ⚠️

Comparison is base (b45d97a) 91.83% compared to head (5456537) 91.81%.

❗ Current head 5456537 differs from pull request most recent head aca7696. Consider uploading reports for the commit aca7696 to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1469      +/-   ##
==========================================
- Coverage   91.83%   91.81%   -0.03%     
==========================================
  Files         655      657       +2     
  Lines       22941    23007      +66     
==========================================
+ Hits        21069    21125      +56     
- Misses       1872     1882      +10     
Impacted Files Coverage Δ
src/common/types/types.cpp 79.61% <ø> (ø)
src/include/storage/copier/table_copier.h 100.00% <ø> (ø)
...rc/include/storage/storage_structure/in_mem_file.h 100.00% <ø> (ø)
src/storage/copier/table_copier.cpp 85.71% <ø> (ø)
src/storage/copier/node_copier.cpp 89.69% <89.02%> (-0.89%) ⬇️
...e/in_mem_storage_structure/in_mem_column_chunk.cpp 90.00% <90.00%> (ø)
...rage/in_mem_storage_structure/in_mem_node_column.h 92.00% <92.00%> (ø)
src/include/storage/copier/node_copier.h 100.00% <100.00%> (ø)
...age/in_mem_storage_structure/in_mem_column_chunk.h 100.00% <100.00%> (ø)
src/include/storage/storage_info.h 100.00% <100.00%> (ø)
... and 4 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@ray6080 ray6080 merged commit 6fc38d0 into master Apr 17, 2023
@ray6080 ray6080 deleted the copy-rework branch April 17, 2023 00:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants