run tests inside SVSM #120

Freax13 · 2023-10-08T09:42:38Z

This PR makes it possible to run tests inside the SVSM. Simply setting the TEST environment variable before running make will produce an SVSM build that runs tests instead of running the guest.

This is implemented using the nightly custom_test_frameworks feature to specify our own tests runner. This nightly feature is only required when building tests for running inside the SVSM, non-test builds and normal test execution can continue to use a stable toolchain. The #[test] macro expects a crate named test to exist, so a small, no_std compatible subset of the test crate has been copied.

Some tests currently fail when run inside the SVSM (AFAICT mostly for benign reasons related to setup of the allocator and file system). Those tests are currently disabled using the ignore attribute. Fixing those tests is probably best left to separate PRs.

roy-hopkins · 2023-10-09T10:33:35Z

Makefile

@@ -9,7 +9,17 @@ TARGET_PATH="debug"
 endif

 STAGE2_ELF = "target/x86_64-unknown-none/${TARGET_PATH}/stage2"
+ifdef TEST
+KERNEL_ELF = "target/x86_64-unknown-none/${TARGET_PATH}/deps/svsm-0651abaef6a489bb"


I'm not sure the metadata hash can be hardcoded here. I think it can change based on compiler version and other parameters.

I think you're right in that there is no guarantee that the hash will be stable. Cargo doesn't seem to have way to fix the path: rust-lang/cargo#1924). Anecdotally, over the course of development for this pr, I've been using multiple nightly versions and the hash remained the same.

I spotted in when I was trying it out - I was playing around with my own build script that deploys to our test machine and I ended up making the hash value change. Unfortunately I don't know what I did to cause it to change and it reverted back to the original when I tried to reproduce it.

Addressed in ace2c55

Makefile

roy-hopkins · 2023-10-10T08:37:39Z

We don't have a README specific to testing at the moment but there is reference to the unit tests in INSTALL.md. Could you maybe add some instructions on how to build and deploy the test binary in INSTALL.md?

joergroedel · 2023-10-12T12:24:06Z

Thanks for doing this, it looks very promising! Somehow I have issues getting the test binary to build, it can't find the core crate. Any hint on how I can fix this on my side?

Freax13 · 2023-10-12T12:26:47Z

Thanks for doing this, it looks very promising! Somehow I have issues getting the test binary to build, it can't find the core crate. Any hint on how I can fix this on my side?

Sounds like the x86_64-unknown-none target on the nightly toolchain isn't installed. Try

rustup target add --toolchain nightly x86_64-unknown-none

joergroedel · 2023-10-12T15:11:06Z

rustup target add --toolchain nightly x86_64-unknown-none

That works, thanks. I can run the tests now.

joergroedel

In general this looks great. A few things I'd like to see changed:

We can safely assume that the test SVSM binary will run in QEMU, so I'd like to use the QEMU exit device to get exit qemu with a given exit-code. This way running the in-kernel tests can be more easily automated.
Please add a Makefile target to build and run the in-kernel tests.

Is there an easy way to limit certain tests to a single test-runner? So that we run some tests only in user-space and others only with the in-kernel runner?

joergroedel · 2023-10-12T15:18:44Z

src/testing.rs

+
+    info!("All tests passed!");
+
+    request_termination_msr();


I wonder if we can use the qemu exit device here?

Turns out we can: 0ef34e2

Freax13 · 2023-10-15T10:02:25Z

In general this looks great. A few things I'd like to see changed:

We can safely assume that the test SVSM binary will run in QEMU, so I'd like to use the QEMU exit device to get exit qemu with a given exit-code. This way running the in-kernel tests can be more easily automated.

Good idea! Implemented in 0ef34e2.

Please add a Makefile target to build and run the in-kernel tests.

Implemented in 773b3b2. I added a test-in-svsm target. This targets determines the QEMU and OVMF installation paths from environment variables (and errors out if they are missing) to run the tests.

Is there an easy way to limit certain tests to a single test-runner? So that we run some tests only in user-space and others only with the in-kernel runner?

We can use cfg_attr(target_os = "none", ignore) to prevent tests from being run inside the SVSM and vice versa.

joergroedel · 2023-10-18T10:20:30Z

We can use cfg_attr(target_os = "none", ignore) to prevent tests from being run inside the SVSM and vice versa.

A more descriptive way of specifying this would be great. Can this work with defining specific features and making tests depend on them?

Freax13 · 2023-10-18T16:01:46Z

We can use cfg_attr(target_os = "none", ignore) to prevent tests from being run inside the SVSM and vice versa.

A more descriptive way of specifying this would be great. Can this work with defining specific features and making tests depend on them?

Yes, we could either use a feature or just a raw cfg(some_feature_name).

Adds tests that make sure a #VC exception is raised while executing various instructions and that the #VC handler properly handles them via the ghcb. Adds tests for cpuid, msrs, port IO, MMIO, wbinvd, rdtsc, dr7, and vmmcall. Relies on being able to run tests inside the guest VM, e.g. with PR coconut-svsm#120. The #VC handler is not yet implemented, so these tests are all marked with #[should_panic]. See PR coconut-svsm#55 for progress on its implementation. TODO: Remove added dependencies Move the tests to vc.rs or a new file Clean up comments, etc. Move assembly wrapper helper functions to more sensible places Implement page state change (PSC) test

Adds tests that make sure a #VC exception is raised while executing various instructions and that the #VC handler properly handles them via the ghcb. Adds tests for cpuid, msrs, port IO, MMIO, wbinvd, rdtsc, dr7, and vmmcall. Relies on being able to run tests inside the guest VM, e.g. with PR coconut-svsm#120. The #VC handler is not yet implemented, so these tests are all marked with #[should_panic]. See PR coconut-svsm#55 for progress on its implementation. TODO: Remove added dependencies Move the tests to vc.rs or a new file Clean up comments, etc. Move assembly wrapper helper functions to more sensible places Implement page state change (PSC) test Signed-off-by: Adam Dunlap <acdunlap@google.com>

Freax13 · 2023-10-21T10:09:51Z

We can use cfg_attr(target_os = "none", ignore) to prevent tests from being run inside the SVSM and vice versa.

A more descriptive way of specifying this would be great. Can this work with defining specific features and making tests depend on them?

Yes, we could either use a feature or just a raw cfg(some_feature_name).

I added a cfg option test_in_svsm. See d368d23 on how this is used.

Freax13 · 2023-10-21T10:11:19Z

I just rebased onto the latest master branch.

joergroedel · 2023-10-23T10:39:40Z

Made a few updates and stored them in this branch. Can you please look at my changes and include them into this PR?

Freax13 · 2023-10-23T11:06:48Z

Made a few updates and stored them in this branch. Can you please look at my changes and include them into this PR?

Looks good to me, thanks!

Once we'll start running tests in the SVSM, we'll want to use the correct implementations and not the fake ones for tests outside the SVSM. Signed-off-by: Tom Dohrmann <erbse.13@gmx.de>

For one reason or another some of the tests currently fail when run inside the SVSM. Signed-off-by: Tom Dohrmann <erbse.13@gmx.de>

Signed-off-by: Tom Dohrmann <erbse.13@gmx.de>

... and move it to `all`. This way we can use the stage1/stage1.o target with another kernel (e.g. test kernel). Signed-off-by: Tom Dohrmann <erbse.13@gmx.de>

To build and run a kernel that runs the tests use ``` QEMU=/path/to/qemu OVMF=/path/to/firmware/ make test-in-svsm ``` Signed-off-by: Tom Dohrmann <erbse.13@gmx.de>

This is cleaner than `request_terminator_msr` as that MSR is more commonly used to signal errors rather than clean exits. Signed-off-by: Tom Dohrmann <erbse.13@gmx.de>

Signed-off-by: Tom Dohrmann <erbse.13@gmx.de>

Add a helper to print the C-bit position on the current platform. Signed-off-by: Joerg Roedel <jroedel@suse.de>

Instead of having the full QEMU command line in the Makefile, extract the test-run into a separate script. Signed-off-by: Joerg Roedel <jroedel@suse.de>

Freax13 · 2023-10-23T11:08:54Z

I also rebased onto the latest master branch so resolve some conflicts.

joergroedel

Thanks, looks good now.

Adds tests that make sure a #VC exception is raised while executing various instructions and that the #VC handler properly handles them via the ghcb. Adds tests for cpuid, msrs, port IO, MMIO, wbinvd, rdtsc, dr7, and vmmcall. Relies on being able to run tests inside the guest VM, e.g. with PR coconut-svsm#120. The #VC handler is not yet implemented, so these tests are all marked with #[should_panic]. See PR coconut-svsm#55 for progress on its implementation. TODO: Remove added dependencies Move the tests to vc.rs or a new file Clean up comments, etc. Move assembly wrapper helper functions to more sensible places Implement page state change (PSC) test Signed-off-by: Adam Dunlap <acdunlap@google.com>

roy-hopkins reviewed Oct 9, 2023

View reviewed changes

roy-hopkins reviewed Oct 10, 2023

View reviewed changes

Makefile Outdated Show resolved Hide resolved

joergroedel reviewed Oct 12, 2023

View reviewed changes

Freax13 force-pushed the feature/custom-test-framework branch 2 times, most recently from 3ea5805 to f3d1e5e Compare October 15, 2023 09:54

Freax13 force-pushed the feature/custom-test-framework branch from f3d1e5e to b043f5b Compare October 21, 2023 10:08

Freax13 force-pushed the feature/custom-test-framework branch from b043f5b to bed5654 Compare October 21, 2023 10:11

Freax13 and others added 9 commits October 23, 2023 11:08

mm: use target_os = "none" instead of test

eee2f13

Once we'll start running tests in the SVSM, we'll want to use the correct implementations and not the fake ones for tests outside the SVSM. Signed-off-by: Tom Dohrmann <erbse.13@gmx.de>

test: disable some tests inside SVSM

20e7531

For one reason or another some of the tests currently fail when run inside the SVSM. Signed-off-by: Tom Dohrmann <erbse.13@gmx.de>

Makefile: mark test as phony

aecbe8c

Signed-off-by: Tom Dohrmann <erbse.13@gmx.de>

test: add no_std compatibly test crate

251359d

Signed-off-by: Tom Dohrmann <erbse.13@gmx.de>

Makefile: remove kernel.elf dependency for stage1.o

f672c5a

... and move it to `all`. This way we can use the stage1/stage1.o target with another kernel (e.g. test kernel). Signed-off-by: Tom Dohrmann <erbse.13@gmx.de>

test: make it possible to run tests inside SVSM

7c5521f

To build and run a kernel that runs the tests use ``` QEMU=/path/to/qemu OVMF=/path/to/firmware/ make test-in-svsm ``` Signed-off-by: Tom Dohrmann <erbse.13@gmx.de>

test: exit using QEMU debug exit device

ad133e1

This is cleaner than `request_terminator_msr` as that MSR is more commonly used to signal errors rather than clean exits. Signed-off-by: Tom Dohrmann <erbse.13@gmx.de>

docs: document how to run tests inside SVSM

ad967ca

Signed-off-by: Tom Dohrmann <erbse.13@gmx.de>

utils: Add cbit helper

4f7ab7d

Add a helper to print the C-bit position on the current platform. Signed-off-by: Joerg Roedel <jroedel@suse.de>

scripts: Add test-in-svsm.sh helper script

845ec96

Instead of having the full QEMU command line in the Makefile, extract the test-run into a separate script. Signed-off-by: Joerg Roedel <jroedel@suse.de>

Freax13 force-pushed the feature/custom-test-framework branch from aba18a1 to 845ec96 Compare October 23, 2023 11:08

joergroedel approved these changes Oct 23, 2023

View reviewed changes

joergroedel merged commit 6138015 into coconut-svsm:main Oct 23, 2023
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

run tests inside SVSM #120

run tests inside SVSM #120

Freax13 commented Oct 8, 2023 •

edited

Loading

roy-hopkins Oct 9, 2023

Freax13 Oct 9, 2023

roy-hopkins Oct 9, 2023

Freax13 Oct 9, 2023

roy-hopkins commented Oct 10, 2023

joergroedel commented Oct 12, 2023

Freax13 commented Oct 12, 2023

joergroedel commented Oct 12, 2023

joergroedel left a comment

joergroedel Oct 12, 2023

Freax13 Oct 15, 2023 •

edited

Loading

Freax13 commented Oct 15, 2023

joergroedel commented Oct 18, 2023

Freax13 commented Oct 18, 2023

Freax13 commented Oct 21, 2023

Freax13 commented Oct 21, 2023

joergroedel commented Oct 23, 2023

Freax13 commented Oct 23, 2023

Freax13 commented Oct 23, 2023

joergroedel left a comment

run tests inside SVSM #120

run tests inside SVSM #120

Conversation

Freax13 commented Oct 8, 2023 • edited Loading

roy-hopkins Oct 9, 2023

Choose a reason for hiding this comment

Freax13 Oct 9, 2023

Choose a reason for hiding this comment

roy-hopkins Oct 9, 2023

Choose a reason for hiding this comment

Freax13 Oct 9, 2023

Choose a reason for hiding this comment

roy-hopkins commented Oct 10, 2023

joergroedel commented Oct 12, 2023

Freax13 commented Oct 12, 2023

joergroedel commented Oct 12, 2023

joergroedel left a comment

Choose a reason for hiding this comment

joergroedel Oct 12, 2023

Choose a reason for hiding this comment

Freax13 Oct 15, 2023 • edited Loading

Choose a reason for hiding this comment

Freax13 commented Oct 15, 2023

joergroedel commented Oct 18, 2023

Freax13 commented Oct 18, 2023

Freax13 commented Oct 21, 2023

Freax13 commented Oct 21, 2023

joergroedel commented Oct 23, 2023

Freax13 commented Oct 23, 2023

Freax13 commented Oct 23, 2023

joergroedel left a comment

Choose a reason for hiding this comment

Freax13 commented Oct 8, 2023 •

edited

Loading

Freax13 Oct 15, 2023 •

edited

Loading