Unit Tests are Stochastic

[biosafetylvl5]

Unit tests introduced in #444 are stochastic:

               for script_path in script_paths:
                    do_geoips_run = rand() < 0.15
                    test_data_found = False
                    if do_geoips_run:
                        # This script has been randomly selected. Check it's contents
                        # to make sure that the test data for the script actually exists
                        .... # do testing things

this increases run time performance. Which is very clever!

However, it is best practice that unit tests are repeatable, which this practice undermines.

I suggest using pytest marks instead to allow testers to choose to run the fast but random test suites or the slow but comprehensive tests. With marks, users could run pytest -m slow which would run all tests which are decorated with the @pytest.mark.slow decorator.

If this is agreed upon as a good idea, I can write out a plan for development.

[jsolbrig]

From talking with @evrose54, it seems that most of the stochastic tests are gone. The only ones remaining is for testing geoips get plugin and the geoips validate. Rather than using a stochastic test for these, I would like to trim down to a single test to start, then use pytest-cov to determine what code is not covered by that single test. Then add additional hard-coded tests to achieve better coverage.

To test coverage and print the lines that are not covered, use pytest --cov --cov-report term-missing. If you want to call pytest on specific tests, you can specify the file that contains the particular test so you don't have to wait as long when testing your specific unit tests: pytest --cov --cov-report term-missing ./tests/unit_tests/path_to_my_test_file.py.

[biosafetylvl5]

Search for rand(*) to find tests that are still stochastic.

[biosafetylvl5]

• Refactor geoips/plugins/modules/output_checkers/image.py
  • instance 1
  • instance 2
  • instance 3
• Refactor geoips/plugins/modules/output_checkers/netcdf.py
• Refactor geoips/plugins/modules/output_checkers/text.py
• Refactor tests/unit_tests/commandline/test_geoips_describe_plugin.py
  • instance 1
  • instance 2
• Refactor tests/unit_tests/commandline/test_geoips_validate.py