package_tests: Invoke tests according to the specification

[stefwalter]

Lets invoke tests according to the specification in the following
document section 1.7.2

https://fedoraproject.org/wiki/Changes/InvokingTests

The previous code took a lot of the interim state of the tests into account. Now that the tests are landing in dist-git lets do them in the perscribed way.

In particular:

• The tests launch the host and/or inventory that they need
• Provide a TEST_ARTIFACTS
• Invoke the playbook from within the dist-git repository
• Use the tests in dist-git proper
• Pass appropriate Ansible variables
• Use the inventory from the test if present

[adl-bot]

Can one of the admins verify this patch?

[adl-bot]

Can one of the admins verify this patch?

[stefwalter]

Updated after further tweaks.

[johnbieren]

My point with having an ansible one and an image one is that the image one will take a qcow2 as a TEST_SUBJECT. I also created the ansible one to fit the following use case: We already have an atomic host up and running and rebased to our newly created ostree, and we want to target that exact host. Because we create ostrees for every pipeline run, and we do not create qcow2s for every package run, we need to be able to run the testing on a certain ostree, not just the base qcow2. So, for example, the workflow here would be, we create the ostree, we provision an atomic host with our latest qcow2, we rebase to the brand new ostree, we make sure it boots, we then run the upstream tests, then we tear it down. Is there any standard way to do this with standard-test-roles now, that you know of? Last I knew there was not, so I was changing the hosts: localhost to hosts: all in the test playbooks to accomplish this.

[stefwalter]

That's good to know.

One of the intents of the test specification is that the test launches up the host (ie: inventory). This was done for two reasons:

• The test may choose to launch two of them, may choose to reboot the host, destroy it in some way, and so on.

• Many of the tests do things to the host that change the host or destroy it. This is just part of routine testing. The order of tests run should not matter.

• We should be able to broadly scale out invocation of testing based on available resources.

But I think we can match the pipeline flow here. Can you:

• Launch the Atomic Host with an overlay qcow2 (this is what we do in Cockpit CI) that points to the base qcow2 image eg: qemu-img create -q -f qcow2 -o backing_file=/path/to/base.qcow2,backing_fmt=qcow2 /path/to/overlay.qcow2
• Update the OSTree with the new package. This affects the overlay qcow2
• Pass the overlay as the TEST_SUBJECT? eg: export TEST_SUBJECT=/path/to/overlay.qcow2

Do you have a pointer to the code that performs these tasks so I could take a look?

[johnbieren]

Here is the code that provisions the atomic host then rebases to the ostree and make sure it boots: https://github.com/CentOS-PaaS-SIG/ci-pipeline/blob/master/config/Dockerfiles/ostree_boot_image/ostree-boot-image.sh
The workflow you described sounds sane, I just need to check on the specifics of it and make sure it can all happen starting from a container, not bare metal.
The package level tests were part of the ostree-boot-sanity stage. When that stage became a container, the package tests were taken out of it, so I have been working this week on making the package tests its own stage in the pipeline. So, I will most likely get it added in (ideally today) the way I described, and then attempt to improve it to the workflow you described.

[johnbieren]

Early googling makes the workflow you describe seem as easy as well, as you described, but I will have to test it out of course first.