Testing framework v2

[rfdavid]

Testing Framework v2

The new framework uses the following logic:

• e2e_read_test scans a directory or opens a specific test file if it's passed as an argument. After we support write operation, we should change this name.
• A TestCase is created for each .test file. TestParser is responsible for reading and tokenizing the test file
• TestCase uses TestParser to check the tokens (STATEMENT, CHECK_ORDER, GROUP etc)
• If the test file is correct, a TestCase object is created containing a vector of TestStatement. TestStatement is a struct (derived from TestQueryConfig) containing the information about that statement.
• There is no need for a .group file anymore. The .test file must contain a header with the following information: -GROUP, -DATASET and -TEST.
• After a TestCase is created, e2e_read_test will register to GTest.
• TestRunner will be responsible to run the test. It reads the TestCase, prepare the logical plans and executes. TestRunner will check if the result corresponds to the expected in the statement.

This is still a work in progress. It currently supports multiple statements, exceptions and statement blocks.

TODO

• Convert the current .test files to the current format
• Reorganize the test files (for example, merge test files four_hop.test and five_hop.test)

Related to #1521

[codecov]

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.01 🎉

Comparison is base (983c3c2) 91.71% compared to head (426f50e) 91.73%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1548      +/-   ##
==========================================
+ Coverage   91.71%   91.73%   +0.01%     
==========================================
  Files         695      695              
  Lines       25031    25033       +2     
==========================================
+ Hits        22957    22963       +6     
+ Misses       2074     2070       -4     

Impacted Files	Coverage Δ
src/common/file_utils.cpp	69.01% <ø> (-1.88%)	⬇️
src/include/common/file_utils.h	100.00% <ø> (ø)
src/include/main/connection.h	100.00% <ø> (ø)
src/include/main/prepared_statement.h	100.00% <ø> (ø)
src/common/string_utils.cpp	94.11% <100.00%> (+3.20%)	⬆️
src/include/common/string_utils.h	100.00% <100.00%> (ø)

... and 2 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[ray6080]

One general comment is that we'd better differentiate test case names for different test files. For example, tests under tinysnb/function can be named as ArithmeticFunction, BooleanFunction, etc., instead of just as Function for all of them.

[ray6080]

LGTM!
One general comment is that we'd better differentiate test case names for different test files. For example, tests under tinysnb/function can be named as ArithmeticFunction, BooleanFunction, etc., instead of just as Function for all of them.

[rfdavid]

One general comment is that we'd better differentiate test case names for different test files. For example, tests under tinysnb/function can be named as ArithmeticFunction, BooleanFunction, etc., instead of just as Function for all of them.

Sounds good! I added to the issue TODO list and I'll change it in the next PR.

[ray6080]

Remove using namespace kuzu::main;. Our convention is that we never use using namespace in header files, but you can use them in cpp files. In header files, we add the namespace prefix as needed.

[rfdavid]

Should I do the same with test_helper.h as well (we're gonna drop test_helper soon)?

[ray6080]

Up to you. Just keep a note is fine.

[ray6080]

Ditto. Remove using namespace kuzu::planner; here.