[FEATURE] openQA tests for SIG/Security override packages

[tcooper]

Is your feature request related to a problem? Please describe.
Recently the Security SIG began providing overrides for a number of packages in Rocky 9. The plan is to also provide those for Rocky 8 in the future. (wiki)

Theoretically we should be able to test application of those packages to various system configurations in openQA to provide confirmation that these overrides work as expected in our critical configurations.

A base test suite that adds the Security SIG repository to the system under test is needed as a starting point. Further tests can be added to install specific overrides along with tests that can/will verify their correct/desired operation can then also be implemented.

Members of the Security Team should be consulted to define the requirements of such tests and ideally they will be able to contribute tests directly to openQA at the time they are developing the overrides.

[sempervictus]

Since we are still early in this process, now would be the time to define a descriptor format identifying all impacted callee functions (including any signature changes since thatll come back to bite us later via CFI) by the patches applied. This should permit identification of callers and exercising them against the patched targets with their current prototypes and data. Specified targeting for testing, autonomously if we can, to make sure we hit the test conditions every time vs relying on test suites of consumers to maybe hit a patched call target (openqa seems a bit higher level, just started reading) or manually maintaining ever more tests. Extraction of those call targets can quite likely be pulled from the diffs by a script.

[solardiz]

For context, @sempervictus comes here from a discussion around glibc hardening. This can explain the focus of his comments. I think most of them actually don't apply, as we're planning rather conservative changes (within same ABI, so no function signature changes for CFI either) and most of them not to libraries. We're also not planning to replace the whole distro, but just some packages - at least as long as the base distro's focus remains like it was (bug-for-bug with RHEL).

So I think what we actually need are (1) unit tests for the changed behavior (e.g., for the few glibc functions that behave subtly differently [yet within spec] when hardened) perhaps invoked during package build (RPM %check section?) and (2) system-wide test suites that ensure other packages and the system as a whole work as intended (per whatever tests we have or can reasonably come up with) when our override and extra packages are installed (including when not all packages are installed, as specified dependencies permit).

An example bug I'd like us to have detected with automated testing is the recent inadvertent and unspecified and undetected dependency of the hardened openssh-server on systemd-devel. I just happened to have the latter package also installed on my test system, so missed the issue. (This is now fixed, avoiding that dependency.) Our test suite should have ensured sshd worked as intended even on a minimal system install that has only packages pulled due to dependencies of whatever few packages we're testing.

[tcooper]

@solardiz The general approach you describe in item 2 would be possible in openQA with proper definition of the post-install checks that would be desired.

Do you still have copies of the openssh build that had the problems you describe above? It's possible that immediate post-build testing with rpminspect could/would have detected that issue. If you have those packages and/or can recreate them it would be worth investigating as it would be much better to catch issues like that before the packages were even published.

[AlanMarshall]

Thoughts on implementation for comment before I commit any code changes:

Add variable DNF_REPOADD=$repo and extra code in _do_install_and_reboot.pm to enable the selected repo (where $repo=security-common or hpc-common) saved in qcow2 for subsequent serial tests.

This would make either install_minimal_upload or install_default_upload test suites available as a base for ongoing tests.

[solardiz]

Do you still have copies of the openssh build that had the problems you describe above?

I still have my own test builds that should exhibit the same problem. I can provide them to you somewhere.

I am also asking on the Testing channel whether we store the old Peridot-made builds anywhere.