Agents: Support writing a test in bash

[ecpullen]

Running a custom test requires building a docker container

Problem: If a TestSys user wants to test features A, they will need to create a container containing the test they want to run.

Solution: Provide 2-3 new containers that are designed to integrate with bash scripts

Bash Agent

The bash agent will accept an encoded bash script and run it. There are a few possible ways this can be expanded.

• Accept a set of scripts and run each in the order they were supplied
  • Each script would be considered a test
  • Users could specify on fail behavior (keep going or stop)

SSM Bash Agent

The bash agent will accept an encoded bash script and run it over ssm to the provided instances. There are a few possible ways this can be expanded.

• Accept a set of scripts and run each in the order they were supplied
  • Each script would be considered a test
  • Users could specify on fail behavior (keep going or stop)

Workload Bash Test

The bash agent will accept an encoded bash script and run it as a workload test. There are a few possible ways this can be expanded.

• Accept a set of scripts and run each in the order they were supplied
  • Each script would be considered a test
  • Users could specify on fail behavior (keep going or stop)

Do we need all 3?

We will want at least 2 of the provided agents. The workload bash test would be useful for prototyping workload test containers like the Nvidia smoke tests and also testing bottlerocket changes that effect workloads.

The SSM Bash agent will be useful for applying setting changes via api client. This is a slight modification of the OS Test Agent.

The Bash agent will be useful for other testing workflows such as applying manifests to a cluster etc. This agent is not nearly as important as the other 2 and can be left out for now.

SSM Bash Agent-

We can adapt the design from the original OS test agent input so that instead of taking in a list of ssm docs it will take in

enum OsTest{
    SsmDoc(String),
    EncodedBash(String),
}

struct OsTestConfig{
    /// The instances that should be targeted by this agent
    instance_ids: Vec<String>,
    /// The tests that this agent will run
    tests: Vec<OsTest>,
    /// Variables that should be set in the environment before each test is run
    env: BTreeMap<String, String>
}

The applied yaml would look like

configuration:
    tests:
        - ssmDoc:
            <SSM DOC NAME>
        - ssmDoc:
            <SECOND SSM DOC NAME>
        - encodedBash:
            <BASE64 ENCODED BASH SCRIPT>
    env:
        ENV_VAR_1: "an env variable"
        ENV_VAR_2: "another env variable"
    onFail: continue
    instanceIds: ${ec2provider.instanceIds}

By accepting env variables as well, a user has full control over their testing. Specifying an OnFail behavior will tell the agent how to respond to a failure. For each test, an exit 0 will distinguish a successful test run.

Workload Bash Agent-

Currently the workload agent accepts a set of WorkloadTest

#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct WorkloadTest {
    pub name: String,
    pub image: String,
    #[serde(default)]
    pub gpu: bool,
}

#[derive(Debug, Clone, Default, Serialize, Deserialize, Configuration, Builder)]
#[serde(rename_all = "camelCase")]
#[crd("Test")]
pub struct WorkloadConfig {
    pub kubeconfig_base64: String,
    pub tests: Vec<WorkloadTest>,
    pub assume_role: Option<String>,
}

We would change this to also accept env variables

#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct WorkloadTest {
    pub name: String,
    pub image: String,
    #[serde(default)]
    pub gpu: bool,
    pub env: BTreeMap<String, String>,
}

The custom workload agent would take the value of ENCODED_BASH and base64 decode it to a bash file and run it.

How much work is it going to take?

The SSM agent should be pretty straight forward, workload bash agent will be especially easy to implement. The complication of these agents will be the use interaction with cargo make test.

cargo make test with the Workload Bash Agent

Currently, users can run a workload test by adding all of the workloads to the workload map

workloads = { <test-name> = <test ctr uri>, <other test> = <another uri> }

Users will continue to list their workloads that way. Before creating the WorkloadTest for each entry, if the test container uri is a file within the TestSys tests dir, that file will be encoded and added as an env variable and the bash workload agent will be used.

workloads = { myBashTest = my_test.sh, myOtherTest = my_test }

tests dir:
tests
|- shared
|   |- my_test.sh
|   |- workloads
|           |- my_test.sh

The snippet above shows accepted file locations and acceptable ways of describing the path (with or without a file extension).

cargo make test with the SSM Bash Agent

Currently, users can run a workload test by adding all of the workloads to the workload map

workloads = { <test-name> = <test ctr uri>, <other test> = <another uri> }

We will define ssm docs/os tests in a similar way.

os-tests = [my-ssm-doc, my-other-ssm-doc, my_bash_test.sh, my_other_bash_test] 

Users will continue to list their SSM docs/bash scripts that way. Before creating the SSMTest for each entry, if the ssm doc uri is a file within the TestSys tests dir, that file will be encoded and added as a OsTest::EncodedBash() variant.

os-tests = { my_bash_test.sh, my_other_bash_test }

tests dir:
tests
|- shared
|   |- my_bash_test.sh
|   |- os-tests
|           |- my_other_bash_test.sh

The snippet above shows accepted file locations and acceptable ways of describing the path (with or without a file extension).

Additionally, we will add a new dev configuration value called env.

[aws.dev]
env = { MY_ENV_VAR = "var1", MY_OTHER_ENV_VAR = "var2" }

All values from the env table will be added to the environment for all os-tests and workload tests.

[cbgbt]

For each test, an exit 0 will distinguish a successful test run.

For other bash-based test runners I've seen, the "success" exit code is configurable, and usually a specific non-zero exit code is selected by default as success (I like 42).

The reason is that it's very easy for a bash program to exit early with a 0 status code without a lot of careful error checking.

[ecpullen]

The reason is that it's very easy for a bash program to exit early with a 0 status code without a lot of careful error checking.

I never considered that. I think that makes a lot of sense.