Skip to content

Tool to run tests in many (emulated) devices at once

License

Notifications You must be signed in to change notification settings

datawire/testbench

Repository files navigation

Datawire testbench --- Run tests in many environments

Installation

Requirements:

  • Python 3.6 or better
  • Python setuptools
  • QEMU
  • Docker
  • GNU Make as /usr/bin/make
  • OVMF
  • kubernaut
  • Only runs on x86_64 hosts

Requirements of testbench-mkosi withmount:

  • losetup (part of util-linux)
  • Linux' unshare(CLONE_NEWNS)
  • Probably some linux specific things about mount(8)

If all of the dependencies are present, it should install just like any other Python package:

$ python3 setup.py install

Usage

There are several executables included; however, some of them are helper scripts that you can ignore.

testbench

The main program is testbench.

Usage: testbench [options] CMD='some command that generates TAP on stdout'

options:
  -j N     Run N virtual machines at once (default: 1).  You
           should generally use `-j$(nproc)`.

Operation:

  1. It looks for test environment descriptions at ./environments/*.mkosi
  2. If nescessary, it compiles those descriptions to virtual machine images at ./environments/*.osi
  3. It runs the specified CMD in each of the test environments, storing the output for each at ./environments/*.tap.
  4. It creates ./testbench.html which is a matrix view of each of all of the .tap files.

Step 2 can be slow; but it only needs to happen once for each .mkosi file. Step 3 should be fairly fast. Runs where step 2 has already been completed should take on the scale of (X+20s)*ceil(M/N), where:

  • X is the ammount of time it takes to run the tests natively
  • M is the number of test environments defined in ./environments/
  • N is the -j N flag.

So if your normal make test takes 3s, and you have 12 test environments, and have 8 cores available, testbench -j$(nproc) should take around (3s+20s)*ceil(12/8) = 23s*2 = 46s.

testbench-mkosi

testbench-mkosi is a fork of mkosi. It is used by testbench internally, but is also useful for entering a virtual machine to investigate failed tests. Read not-updated-for-the-fork docs at [./README.mkosi.md][].

You can interactively launch a test environment (for tight-loop) development with:

$ testbench-mkosi \
    --output   ./environments/ENVNAME.tap.osi \
    --defaults ./environments/ENVNAME.mkosi \
    qemu

Format of .mkosi test environment descriptions

I fibbed a little bit. Test environment descriptions can actualy be multiple files.

  • ./environments/ENVNAME.mkosi (required) base INI-ish description
  • ./environments/ENVNAME.postinst (optional) shell script to run after base operating system install
  • ./environments/ENVNAME.extra (optional) More files that get overlaid on to filesystem, after install (FIXME: does this happen before or after the postinst script is run?). This may either be a directory (in which case all files will be owned by root when copied), or a tarball (which will preserve the ownership information in the tarball).

For the syntax of the .mkosi file itself, see [./README.mkosi.md][].

Arch Linux example:

[Distribution]
Distribution=arch

[Output]
Bootable=yes

[Partitions]
RootSize=4G

[Packages]
Packages=
	docker
	man
	sudo
	networkmanager
WithNetwork=yes

Debian example:

[Distribution]
Distribution=debian

[Output]
Bootable=yes

[Partitions]
RootSize=4G

[Packages]
Packages=
	docker
	man
	sudo
WithNetwork=yes

Related work

About

Tool to run tests in many (emulated) devices at once

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published