Skip to content

Commit

Permalink
test: Enable specifying flaky tests on fips
Browse files Browse the repository at this point in the history
Adds a way to mark a specified test as 'flaky' on fips compliant
systems.

Earlier, the ``tools/test.py`` script supported only 'mode',
'system' and 'arch' for test environment specification. This limits the
ability to specify the behavior of tests and setting pre-determined
behavior of the same on other types of systems. As an example, the
feature request below indicates the need to specify certain tests as
'flaky' on fips compliant systems. It hints at future possibility of a
shared library, which in turn may need a specifier for running tests.

This commit introduces a new item in the ``env`` dict, called ``type``
which defaults to ``simple`` type. It also adds an optional command
line argument ``--type``, which inputs strings. Current functionality
extends to setting ``simple`` or ``fips`` for this ``type`` variable.
However, extending it to further uses is rather simple by adding "if"
conditions at appropriate places in the ``tools/test.py`` script.

PR-URL: #16329
Fixes: #14746
Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
  • Loading branch information
komawar authored and MylesBorins committed Dec 11, 2017
1 parent 298cba6 commit 385653d
Showing 1 changed file with 19 additions and 0 deletions.
19 changes: 19 additions & 0 deletions tools/test.py
Original file line number Diff line number Diff line change
Expand Up @@ -1410,6 +1410,9 @@ def BuildOptions():
result.add_option('--abort-on-timeout',
help='Send SIGABRT instead of SIGTERM to kill processes that time out',
default=False, action="store_true", dest="abort_on_timeout")
result.add_option("--type",
help="Type of build (simple, fips)",
default=None)
return result


Expand Down Expand Up @@ -1559,6 +1562,21 @@ def ArgsToTestPaths(test_root, args, suites):
return paths


def get_env_type(vm, options_type):
if options_type is not None:
env_type = options_type
else:
if "fips" in subprocess.check_output([vm, "-p",
"process.versions.openssl"]):
env_type = "fips"
# NOTE(nikhil): "simple" is the default value for var 'env_type' and should
# be set last if no if/elif matches. If you plan to add more values, use
# 'elif' above.
else:
env_type = "simple"
return env_type


def Main():
parser = BuildOptions()
(options, args) = parser.parse_args()
Expand Down Expand Up @@ -1641,6 +1659,7 @@ def Main():
'mode': mode,
'system': utils.GuessOS(),
'arch': vmArch,
'type': get_env_type(vm, options.type),
}
test_list = root.ListTests([], path, context, arch, mode)
unclassified_tests += test_list
Expand Down

0 comments on commit 385653d

Please sign in to comment.