[develop] Replace shell-based WE2E scripts with python versions (#637)

This PR improves on the new ./run_WE2E_tests.py script (introduced in #558), implementing all the features present in the previous shell-based workflow. Some new files are also introduced for better organization and additional functionality:

* tests/WE2E/utils.py This is a collection of functions used by other scripts, contained here to avoid circular dependencies.
* tests/WE2E/WE2E_summary.py Given an experiment directory or .yaml file, outputs a summary to screen of each experiment, its status, and the number of core hours used. It also prints a summary file with detailed information about each task for each experiment.
* tests/WE2E/print_test_info.py Will print a file WE2E_test_info.txt, very similar to the legacy WE2E_test_info.csv with just a few minor format differences.

Any scripts can be run with the -h argument to print information about all available options (not including utils.py, which is not designed to be run stand-alone).

With this PR, the old superseded shell-based tools are removed.

.cicd/Jenkinsfile

@@ -177,7 +177,7 @@ pipeline {
                         post {
                             always {
                                 // Archive the test log files
-                                sh 'cd "${SRW_WE2E_EXPERIMENT_BASE_DIR}" && tar --create --gzip --verbose --dereference --file "${WORKSPACE}/we2e_test_logs-${SRW_PLATFORM}-${SRW_COMPILER}.tgz" */log.generate_FV3LAM_wflow */log.launch_FV3LAM_wflow */log/*'
+                                sh 'cd "${SRW_WE2E_EXPERIMENT_BASE_DIR}" && tar --create --gzip --verbose --dereference --file "${WORKSPACE}/we2e_test_logs-${SRW_PLATFORM}-${SRW_COMPILER}.tgz" */log.generate_FV3LAM_wflow */log/* ${WORKSPACE}/tests/WE2E/WE2E_tests_*yaml ${WORKSPACE}/tests/WE2E/WE2E_summary*txt ${WORKSPACE}/tests/WE2E/log.*'
                                 // Remove the data sets from the experiments directory to conserve disk space
                                 sh 'find "${SRW_WE2E_EXPERIMENT_BASE_DIR}" -regextype posix-extended -regex "^.*(orog|[0-9]{10})$" -type d | xargs rm -rf'
                                 s3Upload consoleLogLevel: 'INFO', dontSetBuildResultOnFailure: false, dontWaitForConcurrentBuildCompletion: false, entries: [[bucket: 'woc-epic-jenkins-artifacts', excludedFile: '', flatten: false, gzipFiles: false, keepForever: false, managedArtifacts: true, noUploadOnFailure: false, selectedRegion: 'us-east-1', showDirectlyInBrowser: false, sourceFile: 'we2e_test_results-*-*.txt', storageClass: 'STANDARD', uploadFromSlave: false, useServerSideEncryption: false], [bucket: 'woc-epic-jenkins-artifacts', excludedFile: '', flatten: false, gzipFiles: false, keepForever: false, managedArtifacts: true, noUploadOnFailure: false, selectedRegion: 'us-east-1', showDirectlyInBrowser: false, sourceFile: 'we2e_test_logs-*-*.tgz', storageClass: 'STANDARD', uploadFromSlave: false, useServerSideEncryption: false]], pluginFailureResultConstraint: 'FAILURE', profileName: 'main', userMetadata: []

.cicd/scripts/srw_test.sh

@@ -38,58 +38,15 @@ else
 fi
 
 cd ${we2e_test_dir}
-./setup_WE2E_tests.sh ${platform} ${SRW_PROJECT} ${SRW_COMPILER} ${test_type} \
-    expt_basedir=${we2e_experiment_base_dir} \
-    opsroot=${nco_dir}
-
-# Run the new run_srw_tests script if the machine is Cheyenne.
-if [[ "${platform}" = "cheyenne" ]]; then
-    cd ${workspace}/ush
-    ./run_srw_tests.py -e=${we2e_experiment_base_dir}
-    cd ${we2e_test_dir}
-fi
-
 # Progress file
 progress_file="${workspace}/we2e_test_results-${platform}-${SRW_COMPILER}.txt"
-
-# Allow the tests to start before checking for status.
-# TODO: Create a parameter that sets the initial start delay.
-if [[ "${platform}" != "cheyenne" ]]; then
-    sleep 300
-fi
-
-# Wait for all tests to complete.
-while true; do
-
-    # Check status of all experiments
-    ./get_expts_status.sh expts_basedir="${we2e_experiment_base_dir}" \
-         verbose="FALSE" | tee ${progress_file}
-
-    # Exit loop only if there are not tests in progress
-    set +e
-    grep -q "Workflow status:  IN PROGRESS" ${progress_file}
-    exit_code=$?
-    set -e
-
-    if [[ $exit_code -ne 0 ]]; then
-       break
-    fi
-
-    # TODO: Create a paremeter that sets the poll frequency.
-    sleep 60
-done
-
-# Allow we2e cron jobs time to complete and clean up themselves
-# TODO: Create parameter that sets the interval for the we2e cron jobs; this
-# value should be some factor of that interval to ensure the cron jobs execute
-# before the workspace is cleaned up.
-if [[ "${platform}" != "cheyenne" ]]; then
-    sleep 600
-fi
+./setup_WE2E_tests.sh ${platform} ${SRW_PROJECT} ${SRW_COMPILER} ${test_type} \
+    --expt_basedir=${we2e_experiment_base_dir} \
+    --opsroot=${nco_dir} | tee ${progress_file}
 
 # Set exit code to number of failures
 set +e
-failures=$(grep "Workflow status:  FAILURE" ${progress_file} | wc -l)
+failures=$(grep " DEAD    " ${progress_file} | wc -l)
 if [[ $failures -ne 0 ]]; then
     failures=1
 fi

.gitignore

@@ -7,12 +7,14 @@ lib/
 share/
 modulefiles/extrn_comp_build/
 sorc/*/
-tests/WE2E/WE2E_test_info.csv
+tests/WE2E/WE2E_tests_*.yaml
 tests/WE2E/*.txt
 tests/WE2E/*.log
+tests/WE2E/log.*
 ush/__pycache__/
 ush/config.yaml
 ush/python_utils/__pycache__/
 ush/*.swp
 
 *.swp
+__pycache__

docs/UsersGuide/source/ConfigWorkflow.rst

@@ -174,7 +174,7 @@ METplus Parameters
 Test Directories
 ----------------------
 
-These directories are used only by the ``run_WE2E_tests.sh`` script, so they are not used unless the user runs a Workflow End-to-End (WE2E) test. Their function corresponds to the same variables without the ``TEST_`` prefix. Users typically should not modify these variables. For any alterations, the logic in the ``run_WE2E_tests.sh`` script would need to be adjusted accordingly.
+These directories are used only by the ``run_WE2E_tests.py`` script, so they are not used unless the user runs a Workflow End-to-End (WE2E) test (see :numref:`Chapter %s <WE2E_tests>`). Their function corresponds to the same variables without the ``TEST_`` prefix. Users typically should not modify these variables. For any alterations, the logic in the ``run_WE2E_tests.py`` script would need to be adjusted accordingly.
 
 ``TEST_EXTRN_MDL_SOURCE_BASEDIR``: (Default: "")
    This parameter allows testing of user-staged files in a known location on a given platform. This path contains a limited dataset and likely will not be useful for most user experiments.