-
Notifications
You must be signed in to change notification settings - Fork 86
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
reader/csv: sniff column name and type #2116
Conversation
9097cb8
to
c8ec87e
Compare
readerConfig->columnTypes.push_back(stringType.copy()); | ||
columns.push_back(createVariable(columnName, stringType)); | ||
readerConfig->columnTypes.push_back(std::move(columnType)); | ||
columns.push_back(createVariable(columnName, columnTypeID)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You don't need to create a variable for columnTypeID since it is only used once. Just inline the call
void addRow(common::DataChunk&, common::column_id_t column); | ||
|
||
template<typename Driver> | ||
bool addRow(Driver&, uint64_t rowNum, common::column_id_t column_count); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add a variable name for Driver&
return; | ||
} | ||
template<typename Driver> | ||
bool BaseCSVReader::addRow(Driver& driver, uint64_t rowNum, column_id_t column) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Inline this function
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## master #2116 +/- ##
==========================================
- Coverage 89.78% 89.76% -0.03%
==========================================
Files 986 988 +2
Lines 35563 35596 +33
==========================================
+ Hits 31932 31953 +21
- Misses 3631 3643 +12
☔ View full report in Codecov by Sentry. |
The format is `COLUMN_NAME:COLUMN_TYPE`. As part of this change, the CSV reader module has been refactored to use "Drivers" for the actual storing of the values. This allows easy implementation of sniffing, header skipping, and parsing, without affecting performance (because it's all compile time) and it also keeps the reader code clean.
c8ec87e
to
5121c62
Compare
The format is
COLUMN_NAME:COLUMN_TYPE
.