Make tests more readable

[Yi-Tseng]

Now we are using multiple layers of for-loop to create combinations of input parameters, for example:

@autocleanup
def doRunTest(self, param1, param2, ...., paramN):
    # run the test with parameters

def runTest(self):
    for param1 in [...]
        for param2 in [...]
            # skip test for some combanitaions
            if param1 == A and param2 == B:
                continue
            for param3 in [....]
                doRunTest(param1, param2, ..., paramN)

See comment below

We might be able to use python decorator to make the test more readable, for example:

@autocleanup
@test_param('param1', .......)
@test_param('param2', .......)
@skip_test_if(param1=A, param2=B)
@test_param('param3', .......)
def doRunTest(self, param1, param2, param3):
    # run the test with parameters

def runTest(self):
    doRunTest()

[ccascone]

That long list of annotations is not very appealing. It looks like we are defining a meta-language, which can easily become unmanageable.

How about we define a common structure for test arguments, say test_args, and functions to generate that? test_args would include things like the pkt type (test_args.pkt_type), vlan configuration (test_args.ig/eg_vlan_tagged), whether the next hop is spine, etc.

for test_args in self.get_test_args(<high level args>):
    print(test_args) # pkt_type=tcp, vlan=tagget->untagged, next_hop_spine=true, ...
    self.doRunTest(test_args)

An example of <high level args> could be:

• spine_only=true to generate arguments that make sense for a spine switch (eg., interfaces always assumed vlan untagged);
• int_report=true to generate args that make sense for a class that checks report generation;
• etc.

[Yi-Tseng]

Having a function to generate all possible test parameters also sounds like a good idea.
The reason I was thinking about using decorators for the function is that it is easier to custom or modify all input parameters.