GraalJS and runner and reporter improvements - performance, streams, config

[ptrthomas]

this is an "umbrella" issue and I may close some of the existing issues as duplicates. there is a refactoring that will go along with this, perhaps the graal migration

• ✅ introduce a scenario stream concept
  • a "generator function" that can produce scenarios at run-time, lazily for best performance
  • split scenarios for parallel execution
  • result streams can be combined to aggregate results
  • can Scenario Outline take a stream as the data-source ? may be hard
  • at the very least support a "generator" as the call argument
• ✅ test-run progress reporting
  • something we've discussed for the karate-runner VS Code plugin project
  • remaining tests progress indicator
  • dump stats to file that VS Code can pick up as it changes
  • move / rename previous report to backup file so it is not lost / overwritten (good to have)
• ✅ improve Embed design to store files and not hold all the bytes of large images in memory
• run in "quiet" mode, no reports and less logs, to enable bots and high-perf use
• ✅ report performance Karate hangs after finishing the test (with large number of inputs) #1245
  • don't write JSON, XML, HTML when instructed
  • can a file-stream be introduced for perf especially when the no of scenarios is huge
  • re-write JUnit XML report to use something SAX-like not DOM, also see https://stackoverflow.com/q/63747415/143475
• ✅ report vs console logging
  • decouple report verbosity and console logger, maybe use command-line / sys-prop 0.9.6.RC2: HTTP requests/responses not logged to console if showLog: false in report configuration #1156
  • request param or sys-prop should switch debug level for mock also
  • improve logging for multipart https://stackoverflow.com/q/64391922/143475
• ❌ runner pipelines (defer to later)
  • use-case to be able to sequence some tags (or scenario streams) to run first
  • and some control to be able to conditionally abort test after that phase
  • if that is too hard, at the very least plan to introduce a karate.abortAll() api to stop the entire test-suite
• ✅ retry
  • re-design the results objects so that we keep track of re-tries and still report "final" results without messing up scenario / pass / fail counts. this should play well with the "scenario streams" above
  • also support being able to "combine" 2 Result instances, so you can use the Runner multiple times in a "suite" but have a single report Merging two results objects into one object #1263
• ✅ user custom doc can appear inline [enhance] inline doc that appears in HTML report #951
• ✅ improve HTML reporting
  • use new GraalJS based engine
  • templating and re-use system, allow users to create custom reports and contribute as plugin / JAR files
• ✅ report dry-run concept and @todo tag
  • the goal is to NOT require a test-management tool (as far as possible :)
  • you should be able to run tests in “dry run” mode. this will give you the same old HTML report showing what features will be run, we will show all steps so you can read the HTML report. note that we now include comments - but of course the actual times, and logs will be missing. everything will pass
  • the useful part of the “dry run” report is the “Tags” section, showing what tags will run. this should suffice as a feature “coverage” report, provided you have a rich set of tags. e.g. @smoke @module=one @module=two etc
• ✅ because graal-js JS / match improvements - e.g. the JS arrow operator will finally be usable
• ✅ improve match failure reporting Enhancement: Display Expected Results for a Contains Only Failure Using Json Formatting #1012
• ✅ remove / deprecate cucumber reports from stand-alone JAR
• ✅ unify CLI com.intuit.karate:karate-netty:0.9.5 not in maven ? #1177 - right now there are 2 for karate-core and karate-netty
• ✅ debug - improve variables display Show debug variables as json (instead of strings) kirksl/karate-runner#68
• ✅ header names should be case-insensitive Behavior of header matching should treat field names as case-insensitive, but not values #1360

[KostasKgr]

Thanks for raising this Peter, You've noted everything I had in mind.

In case it helps I have crafted a simple example similar to what I had used in a project. Checking task manager in windows it consumes 4.2gb of memory with 2000 elements in the dynamic scenario.
https://github.com/KostasKgr/karate-issues/blob/heavy_cpu_memory_on_dynamic_scenario/src/test/java/examples/example.feature

I had also noticed something weird with the xml report, but for the life of me I can't get the xml report to get generated now that I am at my home windows pc, if I manage to reproduce it I will update this comment.

Cheers

[ptrthomas]

@KostasKgr XML report maybe it is this problem someone just reported: https://stackoverflow.com/q/63747415/143475 - I've added it to the master list

[ptrthomas]

I'm going to declare this as done ! have updated the first comment with the 2 pieces we didn't get to. but very happy with all that we were able to accomplish. we are on GraalJS, karate runs on JDK 16 and the code has been cleaned up very effectively

[dersvenhesse]

That's wonderful news, thank you! Is there an ETA for the changes to be released in a new version?

[ptrthomas]

@dersvenhesse please see https://github.com/intuit/karate/wiki/1.0-upgrade-guide - so right now you can test 0.9.9.RC3