Add "Covered by tests" list for every survived mutation in HTML report

[vivganes]

Right now, when a mutation survives, the tests that cover that mutation are not readily available to the user.

If we provide the list of tests that cover the survived mutation, it would pin-point the places where the user can make improvements.

This PR adds this information to the HTML report.

[hcoles]

Thanks for the PR, I think displaying the tests covering each mutant would be a good improvement.

Some work would be required before this could be merged.

The functionality looks to be driven using the 'succeedingTests' collected by the test listener. This is only populated when pitest is in "fullMatrixMode".

FullMatrixMode is a hack added for the benefit of academic researchers, it makes pitest unusably slow for day to day development and is not a feature I wish to build on or encourage developers to activate.

It would require more work, but roughly the same information should be available without using fullMatrix Mode. Pitest maintains a list of which tests cover each mutant based on coverage. This could be used instead.

A smaller cosmetic change would also be required. The List of covering tests for each mutant will be very large in some codebases. The list will therefore need to be visible only after a mouse click.

[vivganes]

Thanks for the detailed feedback, @hcoles

A few observations/questions:

The list will therefore need to be visible only after a mouse click.

Thanks for this suggestion. I didn't think in this perspective. I will make this change also.

This is only populated when pitest is in "fullMatrixMode".

In this PR, I have tweaked the code to generate this for non fullMatrixMode executions also. My testing and the screenshot generated was not using the fullMatrixMode

From your comment, I now realize that mine is probably a 'hacky' implementation in this line:

pitest/pitest-entry/src/main/java/org/pitest/mutationtest/MutationStatusMap.java

Lines 55 to 58 in d317859

	finalStatus = new MutationStatusTestPair(status.getNumberOfTestsRun(),
	status.getStatus(),
	status.getKillingTests(),
	passingTests);

This hacky way enables us to show the covered tests without significantly changing the current code.

However, if you still want me to explore and implement your suggestion of reading it from the place where we have the tests for each mutant, I will start work on that today.

Please let me know what you think.

[hcoles]

If the sucedingTest property has been repurposed so that it now contains covering tests sometimes and sometimes tests the mutant was actually challenged with, this will be very confusing to maintain. What happens when fullMatrixMode is enabled?

I think a seperate coveringTests property is required.

[vivganes]

If the sucedingTest property has been repurposed so that it now contains covering tests sometimes and sometimes tests the mutant was actually challenged with, this will be very confusing to maintain. What happens when fullMatrixMode is enabled?

I think a seperate coveringTests property is required.

Thanks for your input. I will make this change.

[vivganes]

@hcoles I have made all the changes suggested by you. I have also updated the screenshot in the pull request description.

Please let me know if this needs more work.

[hcoles]

Thanks, now merged.

[vivganes]

Thanks for all your inputs @hcoles 🙏

Is there any estimate on when I can get this change in mvn repo? My team could really use this new report features.

[hcoles]

@vivganes I'll release towards the end of this week.

[obscurebyron]

Is there any way the applicable tests could be shown adjacent to the mutation? I just tried this and it's very helpful - but having the tests that apply for a particular mutation, like the screenshot from @vivganes would send its usefulness through the stratosphere, since I am often aiming to tune one small piece, rather than a whole class. Just to double check, the current implementation shows the applicable tests for the entire class, in the report at the bottom under "Tests examined", correct? Regardless, kudos on this excellent work!

[hcoles]

@obscurebyron

The "Tests examined" for whole class have been there for some time. The new addition is a link next to each surviving mutant (in the bottom section where mutants are listed by line number) that expands to show the covering tests.

[obscurebyron]

Yes, I had missed that. Ah, that is absolutely fantastic! Thank you @hcoles and @vivganes