-
Notifications
You must be signed in to change notification settings - Fork 86
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Test Framework: Support CSV to Parquet conversion (#1611)
Convert CSV dataset to PARQUET dataset inside .test files by using: -DATASET PARQUET CSV_TO_PARQUET(dataset)
- Loading branch information
Showing
106 changed files
with
3,836 additions
and
105 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,49 @@ | ||
#include "main/kuzu.h" | ||
#include <arrow/api.h> | ||
|
||
namespace kuzu { | ||
namespace testing { | ||
|
||
// Convert an entire CSV dataset directory to parquet. | ||
// The dataset directory must contain schema and copy files. | ||
class CSVToParquetConverter { | ||
public: | ||
static void convertCSVDatasetToParquet(std::string& dataset); | ||
|
||
private: | ||
struct CopyCommandInfo { | ||
std::string table; | ||
std::string csvFilePath; | ||
std::string parquetFilePath; | ||
bool csvHasHeader; | ||
char delimiter; | ||
}; | ||
|
||
static std::vector<CopyCommandInfo> readCopyCommandsFromCopyCypherFile( | ||
const std::string& dataset); | ||
|
||
static void convertCSVFilesToParquet( | ||
const std::vector<CSVToParquetConverter::CopyCommandInfo>& copyCommands); | ||
|
||
static CopyCommandInfo createCopyCommandInfo( | ||
const std::string& dataset, std::string copyStatement); | ||
|
||
static arrow::Status runCSVToParquetConversion(const std::string& inputFile, | ||
const std::string& outputFile, char delimiter, bool hasHeader); | ||
|
||
static void copySchema( | ||
const std::string& csvDatasetPath, const std::string& parquetDatasetPath); | ||
|
||
static void createCopyFile(const std::string& dataset, | ||
const std::vector<CSVToParquetConverter::CopyCommandInfo>& copyCommands); | ||
|
||
inline static std::string replaceSlashesWithUnderscores(std::string dataset) { | ||
std::replace(dataset.begin(), dataset.end(), '/', '_'); | ||
return dataset; | ||
} | ||
|
||
static std::string extractPath(std::string& str, char delimiter); | ||
}; | ||
|
||
} // namespace testing | ||
} // namespace kuzu |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,5 @@ | ||
-GROUP CopyNodeFromCSVTest | ||
-DATASET copy-test/node/csv | ||
-DATASET CSV copy-test/node/csv | ||
|
||
-- | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,5 @@ | ||
-GROUP CopyNodeFromParquetTest | ||
-DATASET copy-test/node/parquet | ||
-DATASET CSV copy-test/node/parquet | ||
|
||
-- | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,5 @@ | ||
-GROUP DemoDBTest | ||
-DATASET demo-db/csv | ||
-DATASET CSV demo-db/csv | ||
|
||
-- | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,5 @@ | ||
-GROUP DemoDBCreateTest | ||
-DATASET demo-db/csv | ||
-DATASET CSV demo-db/csv | ||
|
||
-- | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,5 @@ | ||
-GROUP DemoDBDeleteTest | ||
-DATASET demo-db/csv | ||
-DATASET CSV demo-db/csv | ||
|
||
-- | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,5 @@ | ||
-GROUP DemoDBTest | ||
-DATASET demo-db/csv | ||
-DATASET CSV demo-db/csv | ||
|
||
-- | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,5 @@ | ||
-GROUP DemoDBTest | ||
-DATASET demo-db/parquet | ||
-DATASET PARQUET demo-db/parquet | ||
|
||
-- | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,5 @@ | ||
-GROUP DemoDBTest | ||
-DATASET demo-db/parquet | ||
-DATASET PARQUET demo-db/parquet | ||
|
||
-- | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,5 @@ | ||
-GROUP DemoDBSetAndCopyTest | ||
-DATASET demo-db/csv | ||
-DATASET CSV demo-db/csv | ||
|
||
-- | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,5 @@ | ||
-GROUP BinderErrorTest | ||
-DATASET tinysnb | ||
-DATASET CSV tinysnb | ||
|
||
-- | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,5 @@ | ||
-GROUP CatalogErrorTest | ||
-DATASET tinysnb | ||
-DATASET CSV tinysnb | ||
|
||
-- | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,5 @@ | ||
-GROUP SyntaxErrorTest | ||
-DATASET dummy | ||
-DATASET CSV dummy | ||
|
||
-- | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,5 @@ | ||
-GROUP SyntaxErrorTest | ||
-DATASET dummy | ||
-DATASET CSV dummy | ||
|
||
-- | ||
|
||
|
Oops, something went wrong.