Scan and parse test files only once

[rfdavid]

This PR implements scanning and parsing the test files only once instead of every e2e_test run.

closes #1651

[codecov]

Codecov Report

Patch coverage has no change and project coverage change: -0.03 ⚠️

Comparison is base (322dab6) 91.49% compared to head (3df06f2) 91.46%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1654      +/-   ##
==========================================
- Coverage   91.49%   91.46%   -0.03%     
==========================================
  Files         727      726       -1     
  Lines       26386    26411      +25     
==========================================
+ Hits        24141    24158      +17     
- Misses       2245     2253       +8     

see 17 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[ray6080]

I wonder if we should just not rely on ctest, but directly run e2e_test binary?

[rfdavid]

I wonder if we should just not rely on ctest, but directly run e2e_test binary?

Yes, we can directly run e2e_test binary by passing the test file or directory. We use ctest utility to organize what tests to run and provide a nice output. If we are going to run e2e_test directly, we will have to create a minimal utility to organize what tests to run, perform e2e_test passing the file, capture the output, summarize and print the results. We can have more control. It might be a good alternative.