Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Read NPY chunk at a time #1697

Merged
merged 1 commit into from
Jun 21, 2023
Merged

Read NPY chunk at a time #1697

merged 1 commit into from
Jun 21, 2023

Conversation

aziz-mu
Copy link
Contributor

@aziz-mu aziz-mu commented Jun 20, 2023

This PR solves (1) from the issue #1670, although some design changes were made from the issue:
(i) We are keeping mmap for now as it simplifies reading the header
(ii) We are using arrow::RecordBatch to match the design of ReadFile

@aziz-mu aziz-mu requested a review from ray6080 June 20, 2023 15:36
@codecov
Copy link

codecov bot commented Jun 20, 2023

Codecov Report

Patch coverage: 100.00% and no project coverage change.

Comparison is base (251d978) 91.22% compared to head (dc8745b) 91.22%.

❗ Current head dc8745b differs from pull request most recent head 94d01a1. Consider uploading reports for the commit 94d01a1 to get more accurate results

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1697   +/-   ##
=======================================
  Coverage   91.22%   91.22%           
=======================================
  Files         737      737           
  Lines       26987    26994    +7     
=======================================
+ Hits        24619    24626    +7     
  Misses       2368     2368           
Impacted Files Coverage Δ
src/include/storage/copier/npy_reader.h 100.00% <ø> (ø)
...age/in_mem_storage_structure/in_mem_column_chunk.h 100.00% <ø> (ø)
src/storage/copier/node_copier.cpp 62.93% <100.00%> (+0.65%) ⬆️
src/storage/copier/npy_reader.cpp 90.74% <100.00%> (+2.10%) ⬆️
...e/in_mem_storage_structure/in_mem_column_chunk.cpp 81.62% <100.00%> (+0.16%) ⬆️

... and 8 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

src/storage/copier/npy_reader.cpp Outdated Show resolved Hide resolved
src/storage/copier/npy_reader.cpp Outdated Show resolved Hide resolved
@aziz-mu aziz-mu merged commit 3c5f892 into master Jun 21, 2023
8 checks passed
@aziz-mu aziz-mu deleted the npy-reader-updates branch June 21, 2023 16:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants