You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We support npy as a file format to be copied into node tables, specifically into fixed size array columns.
Currently, npy copy goes through NodeCopyExecutor, which is separate from our query processor. While copying of csv and parquet files goes through the query processor (in the issue title, I call it "COPY pipeline").
For simplicity of code, we should move npy copy to go through the query processor too.
Specifically, we need to remove NodeCopyExecutor, and move the logic of copying npy files to CopyNode and ReadFile operators.
Two main issues need to be addressed for this change:
Change NpyReader to read a DataChunk at a time, so it can fit into ReadFile opeartor. The use of mmap can be removed since we read sequentially, which should be fine from the perspective of IO performance.
NPY copy follows the syntax of COPY BY COLUMN, thus one file corresponds to one column, which is quite different from CSV and Parquet files. CSV and Parquet files cover all columns in a single file. So CopyNode should be aware of that npy is copying a DataChunk into one column at a time, instead of all columns.
The text was updated successfully, but these errors were encountered:
We support npy as a file format to be copied into node tables, specifically into fixed size array columns.
Currently, npy copy goes through
NodeCopyExecutor
, which is separate from our query processor. While copying of csv and parquet files goes through the query processor (in the issue title, I call it "COPY pipeline").For simplicity of code, we should move npy copy to go through the query processor too.
Specifically, we need to remove
NodeCopyExecutor
, and move the logic of copying npy files toCopyNode
andReadFile
operators.Two main issues need to be addressed for this change:
NpyReader
to read a DataChunk at a time, so it can fit intoReadFile
opeartor. The use ofmmap
can be removed since we read sequentially, which should be fine from the perspective of IO performance.COPY BY COLUMN
, thus one file corresponds to one column, which is quite different from CSV and Parquet files. CSV and Parquet files cover all columns in a single file. SoCopyNode
should be aware of that npy is copying a DataChunk into one column at a time, instead of all columns.The text was updated successfully, but these errors were encountered: