Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Copy to csv rework #2242

Merged
merged 1 commit into from
Oct 20, 2023
Merged

Copy to csv rework #2242

merged 1 commit into from
Oct 20, 2023

Conversation

acquamarin
Copy link
Collaborator

@acquamarin acquamarin commented Oct 19, 2023

This PR improves the existing copy-to-csv:

  1. Allow copy to csv to be executed in parallel.
  2. Avoid unnecessary flattens on copy to vectors.

E2E performance on copy ldbc-100 to csv file:
60452.38ms (M1 Max, 32GB RAM)

@codecov
Copy link

codecov bot commented Oct 19, 2023

Codecov Report

Attention: 4 lines in your changes are missing coverage. Please review.

Comparison is base (d81c628) 89.69% compared to head (4bed782) 89.69%.
Report is 5 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #2242   +/-   ##
=======================================
  Coverage   89.69%   89.69%           
=======================================
  Files        1012     1013    +1     
  Lines       35862    35846   -16     
=======================================
- Hits        32167    32153   -14     
+ Misses       3695     3693    -2     
Files Coverage Δ
src/common/serializer/buffered_serializer.cpp 92.85% <100.00%> (ø)
src/common/type_utils.cpp 86.07% <100.00%> (+16.26%) ⬆️
src/common/types/types.cpp 90.82% <100.00%> (ø)
src/function/vector_cast_functions.cpp 80.45% <100.00%> (+1.61%) ⬆️
...rc/include/common/serializer/buffered_serializer.h 66.66% <100.00%> (+33.33%) ⬆️
src/include/common/type_utils.h 100.00% <ø> (ø)
src/include/common/types/types.h 100.00% <ø> (ø)
src/include/function/cast/cast_functions.h 99.08% <100.00%> (-0.02%) ⬇️
src/include/function/cast/vector_cast_functions.h 95.65% <ø> (ø)
...rc/include/processor/operator/persistent/copy_to.h 100.00% <100.00%> (ø)
... and 9 more

... and 17 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

@andyfengHKU andyfengHKU left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Take a look at coverage. I feel it should be covered but the report says no.

src/include/common/constants.h Show resolved Hide resolved
src/include/function/cast/cast_functions.h Outdated Show resolved Hide resolved
src/processor/operator/physical_operator.cpp Show resolved Hide resolved
src/processor/operator/persistent/copy_to_parquet.cpp Outdated Show resolved Hide resolved
src/common/types/types.cpp Show resolved Hide resolved
@acquamarin acquamarin merged commit cf28bdd into master Oct 20, 2023
12 checks passed
@acquamarin acquamarin deleted the copy-to-csv branch October 20, 2023 14:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants