Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change hash index concurrent build #1564

Merged
merged 1 commit into from
May 24, 2023
Merged

Change hash index concurrent build #1564

merged 1 commit into from
May 24, 2023

Conversation

ray6080
Copy link
Contributor

@ray6080 ray6080 commented May 24, 2023

Our hash index builder is not scalable. We need to rework on that.
Currently, I'm putting locks each time a thread flushes into the index. This can improve copying. On my local M1 Air laptop, it improves copying of ldbc10 comment csv files from ~28s to ~12s, and parquet files from ~22.4s to 5.6s.

@codecov
Copy link

codecov bot commented May 24, 2023

Codecov Report

Patch coverage: 100.00% and project coverage change: -0.01 ⚠️

Comparison is base (91204de) 91.73% compared to head (43e0393) 91.72%.

❗ Current head 43e0393 differs from pull request most recent head faa5756. Consider uploading reports for the commit faa5756 to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1564      +/-   ##
==========================================
- Coverage   91.73%   91.72%   -0.01%     
==========================================
  Files         698      698              
  Lines       25128    25113      -15     
==========================================
- Hits        23050    23034      -16     
- Misses       2078     2079       +1     
Impacted Files Coverage Δ
src/storage/copier/node_copy_executor.cpp 91.30% <ø> (ø)
src/storage/index/hash_index_builder.cpp 89.58% <ø> (-0.99%) ⬇️
src/binder/bound_statement_result.cpp 92.30% <100.00%> (ø)
src/include/storage/index/hash_index_builder.h 79.16% <100.00%> (-2.66%) ⬇️
src/include/storage/storage_info.h 100.00% <100.00%> (ø)
src/storage/copier/node_copier.cpp 91.55% <100.00%> (+0.11%) ⬆️

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@ray6080 ray6080 merged commit 253ad17 into master May 24, 2023
3 checks passed
@ray6080 ray6080 deleted the copy-rework branch May 24, 2023 16:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants