Testing

Triggering Tests in GitHub Pull Requests (short)

First, do @susebot run teuthology (triggers a single "generalist" integration test job).

If that passes, do @susebot run teuthology suse/tier2 (triggers a suite of integration tests)

To run both of these at once, use @susebot run teuthology suse.

Read the rest of this wiki page for more details on this topic.

What Are These Tests? What Do They Do?

The tests discussed here are integration tests. Each test runs in its own set of VMs and deploys a Ceph cluster by running DeepSea stages.

The tests are organized into 'tiers', which are numbered in increasing order to express how much the tests inside stress the underlying testing infrastructure.

Tier 0)

Tier 0 contains the most basic test one can imagine (see the definition of "smoke test") for DeepSea, demonstrating its capability to deploy a minimal Ceph cluster.

Tier 1)

Spins up a single-node cluster with mds and rgw roles, checks for its health/sanity (*), and runs the existing functional tests (**) on it.

(*) All tests are based on a health-ok test that just spins up a cluster and checks for it's health/sanity. This includes checking that "ceph -s" shows HEALTH_OK. In addition, for clusters with mds role it includes mounting cephfs and touching a file. Clusters with roles, such as igw, mds, nfs-ganesha, rgw, etc., are subjected to similar, minimal "sanity" tests of the respective components.

(**) The functional tests run the ceph.functests.1node orchestration on the test cluster. That includes tests for the following DeepSea operations:

• restart (mds, rgw, mon, mgr)
• keyring creation
• quiescent
• apparmor
• tuned
• remove
• replace
• openstack

Tier 2)

Runs deployment tests, with minimal validation, for all major components:

• mds
• igw (SES5-only)
• nfs-ganesha
• openattic (SES5-only)
• rgw

The idea is for Tier 2 to hold only those tests that are known to reliably produce meaningful results within a relatively short time. Thus, the tests typically run on single-node clusters, with a separate client node where necessary.

Tier 3)

Currently:

• rbd
• stage.5 (Removal of a node )

Tier 3 holds a set of tests that provide more in-depth coverage of various components and features, either of Ceph or of DeepSea itself.

Tier 4)

Tier 4 currently holds tests that are unreliable, use up lots of testing infrastructure resources, or are experimental or otherwise under development.

A prime candidate for this tier are the profile-deployment tests. These tests are designed to test all possible manifestations of OSD configurations (bluestore, filestore, dedicated wal, dedicated db, etc.). A list of those can be found here.

Another Tier 4 test is the migration functional tests. The objective for those tests is to test Deepsea's migration feature in all possible combinations. (Filestore to Bluestore, Bluestore to Filestore, etc.) Due to issues with the VMs we are testing on ( 2018-08-27 ) we expect unpredictable failures in some of the tests.

Triggering Tests in GitHub Pull Requests (long)

If you want to run tests against your GitHub pull request ("PR"), you can leverage Jenkins+Teuthology for this.

You can leave a comment in the PR to trigger the tests. @susebot listens to specific commands such as:

@susebot run teuthology

This would run the Tier 1 test on the PR - this is sufficient in many cases, and should be the first integration test you run in any case.

The following commands (which are identical) trigger the Tier 2 tests:

@susebot run teuthology suse/tier2 @susebot run teuthology deepsea/tier2

(suse/tier2 is just a symlink to deepsea/tier2)

You can be more fine-grained if you only want to run specific Tier 2 tests. For example you can write:

@susebot run teuthology deepsea/tier2/health-ok

Triggers only the health-ok tests.

@susebot run teuthology deepsea/tier2/rgw

Triggers an rgw test suite.

You can also trigger Tier 3 tests:

@susebot run teuthology deepsea/tier3/all-osd-configs

Runs Tier 3 all-osd-config tests.

Where the test code is maintained

The test code consists of the "deepsea.py" teuthology task and associated YAML files. These are maintained in SUSE/ceph.git in the ses6 and ses5 branches.

In some cases, DeepSea PRs may require changes to the test code. To facilitate this, the DeepSea PR CI uses different branches - ses6-deepsea-pr and ses5-deepsea-pr. That way, when a disruptive PR is merged, the test code changes can be merged to these branches to allow DeepSea development to go forward. Since the SES (Devel/Product) CI uses the ses6 and ses5 branches, it is not affected by these changes.

Finally, whenever a new DeepSea release is cut and pushed to Devel/Product, any test code changes in ses6-deepsea-pr need to be merged into ses6 at the same time to keep the Devel/Product CI from breaking.