Merge remote-tracking branch 'upstream/develop' into merra2

* upstream/develop:
  Updatetemplate; Fix model slowness when using threading; Update WW3 for porting and threading issues (ufs-community#383)
  update MOM6 to GFDL 20210308 main branch commit (ufs-community#458)
  Regional inlinepost (ufs-community#364)
  correct benchmark diag_tables for coupled model configurations; move bm_ic directory out of inputdata directory; use aws ec2 for CI test; auto-rt fixes (ufs-community#426)

.github/pull_request_template.md

@@ -1,29 +1,46 @@
+# PR Checklist
+
+- [ ] Ths PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR. Please consult the ufs-weather-model [wiki](https://github.com/ufs-community/ufs-weather-model/wiki/Making-code-changes-in-the-UFS-weather-model-and-its-subcomponents) if you are unsure how to do this.
+
+- [ ] This PR has been tested using a branch which is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR
+
+- [ ] An Issue describing the work contained in this PR has been created either in the subcomponent(s) or in the ufs-weather-model. The Issue should be created in the repository that is most relevant to the changes in contained in the PR. The Issue and the dependent sub-component PR 
+are specified below.
+
+- [ ] If new or updated input data is required by this PR, it is clearly stated in the text of the PR.
+
+## Instructions: All subsequent sections of text should be filled in as appropriate.
+
+The information provided below allows the code managers to understand the changes relevant to this PR, whether those changes are in the ufs-weather-model repository or in a subcomponent repository. Ufs-weather-model code managers will use the information provided to add any applicable labels, assign reviewers and place it in the Commit Queue. Once the PR is in the Commit Queue, it is the PR owner's responsiblity to keep the PR up-to-date with the develop branch of ufs-weather-model. 
+
 ## Description
 
-(Instructions: this, and all subsequent sections of text should be removed and filled in as appropriate.)
-Provide a detailed description of what this PR does.
-What bug does it fix, or what feature does it add?
-Is a change of answers expected from this PR?
-Are any library updates included in this PR (modulefiles etc.)?
+Provide a detailed description of what this PR does. What bug does it fix, or what feature does it add? Is a change of answers expected from this PR? Are any library updates included in this PR (modulefiles etc.)?
 
 ### Issue(s) addressed
 
 Link the issues to be closed with this PR, whether in this repository, or in another repository.
-(Remember, issues should always be created before starting work on a PR branch!)
+(Remember, issues must always be created before starting work on a PR branch!) 
 - fixes #<issue_number>
 - fixes noaa-emc/fv3atm/issues/<issue_number>
 
 ## Testing
 
-How were these changes tested?
-What compilers / HPCs was it tested with?
-Are the changes covered by regression tests? (If not, why? Do new tests need to be added?)
-Have regression tests and unit tests (utests) been run? On which platforms and with which compilers? (Note that unit tests can only be run on tier-1 platforms)
+How were these changes tested? What compilers / HPCs was it tested with? Are the changes covered by regression tests? (If not, why? Do new tests need to be added?) Have regression tests and unit tests (utests) been run? On which platforms and with which compilers? (Note that unit tests can only be run on tier-1 platforms)
+
+- [ ] hera.intel
+- [ ] hera.gnu
+- [ ] orion.intel
+- [ ] cheyenne.intel 
+- [ ] cheyenne.gnu
+- [ ] gaea.intel 
+- [ ] jet.intel
+- [ ] wcoss_cray
+- [ ] wcoss_dell_p3
 
 ## Dependencies
 
-If testing this branch requires non-default branches in other repositories, list them.
-Those branches should have matching names (ideally)
+If testing this branch requires non-default branches in other repositories, list them. Those branches should have matching names (ideally).
 
 Do PRs in upstream repositories need to be merged first?
 If so add the "waiting for other repos" label and list the upstream PRs

.github/workflows/main.yml → .github/workflows/build_test.yml

@@ -1,17 +1,15 @@
-name: Pull Request Tests
+name: Pull request tests
 
 on:
   push:
-    branches:
-    - develop
+    branches: ['develop']
   pull_request:
-    branches:
-    - develop
+    branches: ['develop']
 
 jobs:
   setup:
     name: Set up
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-20.04
 
     outputs:
       tn: ${{ steps.parse.outputs.tn }}
@@ -46,7 +44,7 @@ jobs:
   build:
     name: Build (${{ matrix.bld_set }})
     needs: setup
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-20.04
 
     strategy:
       fail-fast: false
@@ -85,8 +83,7 @@ jobs:
   utest:
     name: Unit test (${{ needs.setup.outputs.tn }}, ${{ matrix.test_set }})
     needs: [setup,build]
-    runs-on: ubuntu-latest
-    #runs-on: self-hosted
+    runs-on: self-hosted
 
     strategy:
       fail-fast: false
@@ -101,7 +98,7 @@ jobs:
     - name: Prepare artifacts
       run: |
         tar xvjf artifact.tar.bz2 && rm -f artifact.tar.bz2
-        sudo docker load --input ${{ needs.setup.outputs.img }}.tar.gz && rm -f ${{ needs.setup.outputs.img }}.tar.gz
+        docker load --input ${{ needs.setup.outputs.img }}.tar.gz && rm -f ${{ needs.setup.outputs.img }}.tar.gz
 
     - name: Run utest
       run: ./ci.sh -n ${{ needs.setup.outputs.tn }} -r ${{ matrix.test_set }}
@@ -117,5 +114,7 @@ jobs:
       if: ${{ always() }}
       run: |
         rm -f ci.sh ci.test
-        sudo docker rm  my-container && sudo docker rmi ${{ needs.setup.outputs.img }}:latest
-        sudo docker volume rm DataVolume
+        docker stop my-container && docker rm  my-container && docker rmi ${{ needs.setup.outputs.img }}:latest
+        docker volume rm DataVolume
+        #docker rmi minsukjinoaa/fv3-input-data:input-data-20210115
+        rm -f memory_stat

.github/workflows/manage.yml → .github/workflows/manage_workflows.yml

@@ -2,34 +2,36 @@ name: Manage workflows
 
 on:
   workflow_run:
-    workflows: ["Pull Request Tests"]
+    workflows: ["Pull request tests"]
     types:
       - requested
 
 jobs:
   job1:
-    name: Job 1
-    runs-on: ubuntu-latest
+    name: Cancel workflows
+    runs-on: ubuntu-20.04
 
     steps:
       - name: Checkout codes
         uses: actions/checkout@v2
 
-      - name: Check if skip-ci is requested
+      - name: Check if run-ci is requested
         run: |
+          sleep 40
           cd ${GITHUB_WORKSPACE}/tests/ci
           repo="${GITHUB_API_URL}/repos/${GITHUB_REPOSITORY}/actions/runs"
           tr_id=$(cat ${GITHUB_EVENT_PATH} | ./json_helper.py get_trigger_id)
           tr_br=$(cat ${GITHUB_EVENT_PATH} | ./json_helper.py get_trigger_br)
-          check=$(cat ${GITHUB_EVENT_PATH} | ./json_helper.py check_skip)
+          check=$(cat ${GITHUB_EVENT_PATH} | ./json_helper.py check_run)
           echo "TRIGGER_ID=${tr_id}" >> $GITHUB_ENV
           echo "TRIGGER_BR=${tr_br}" >> $GITHUB_ENV
-          echo "skip-ci: ${check}"
-          if [[ $check == yes ]]; then
-            echo "skip-ci is requested"
+          echo "run-ci: ${check}"
+          if [[ $check == no ]]; then
+            echo "run-ci is not requested"
             echo "CURR_JOB=cancelled" >> $GITHUB_ENV
             curl -X POST -H "Authorization: token ${{ secrets.GITHUB_TOKEN }}" -H "Accept: application/vnd.github.v3+json" ${repo}/$tr_id/cancel
           else
+            echo "run-ci is requested"
             echo "CURR_JOB=running" >> $GITHUB_ENV
           fi
 

.github/workflows/start_runners.yml

@@ -0,0 +1,33 @@
+name: Start runners
+
+on:
+  workflow_run:
+    workflows: ["Pull request tests"]
+    types:
+      - requested
+
+jobs:
+  job1:
+    name: Start AWS runners
+    runs-on: ubuntu-20.04
+
+    steps:
+    - name: Check out codes
+      uses: actions/checkout@v2
+
+    - name: Configure AWS
+      uses: aws-actions/configure-aws-credentials@v1
+      with:
+        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+        aws-region: us-east-1
+
+    - name: Check the status of build and start self-hosted runners
+      env:
+        aws_instance_id: ${{ secrets.AWS_INSTANCE_ID }}
+      run: |
+        cd ${GITHUB_WORKSPACE}/tests/ci
+        conclusion=$(cat ${GITHUB_EVENT_PATH} | ./build_status_check.py)
+        if [[ $conclusion == "success" ]]; then
+          aws ec2 start-instances --instance-ids $aws_instance_id
+        fi

.github/workflows/stop_runners.yml

@@ -0,0 +1,25 @@
+name: Stop runners
+
+on:
+  workflow_run:
+    workflows: ["Pull request tests"]
+    types:
+      - completed
+
+jobs:
+  job1:
+    name: Stop AWS runners
+    runs-on: ubuntu-20.04
+
+    steps:
+    - name: Configure AWS
+      uses: aws-actions/configure-aws-credentials@v1
+      with:
+        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+        aws-region: us-east-1
+
+    - name: Stop self-hosted runners
+      env:
+        aws_instance_id: ${{ secrets.AWS_INSTANCE_ID }}
+      run: aws ec2 stop-instances --instance-ids $aws_instance_id

CMakeLists.txt

@@ -204,13 +204,7 @@ endif()
 ###############################################################################
 if(WW3)
   if(CMAKE_Platform)
-    if(${CMAKE_Platform} STREQUAL "hera.intel")
-      set(WW3_COMP "hera")
-    elseif(${CMAKE_Platform} STREQUAL "orion.intel")
-      set(WW3_COMP "orion")
-    elseif(${CMAKE_Platform} STREQUAL "cheyenne.intel")
-      set(WW3_COMP "cheyenne")
-    elseif(${CMAKE_Platform} STREQUAL "wcoss2")
+    if(${CMAKE_Platform} STREQUAL "wcoss2")
       set(WW3_COMP "wcoss_cray")
     elseif(${CMAKE_Platform} MATCHES "linux*" OR ${CMAKE_Platform} MATCHES "macosx*")
       if (CMAKE_Fortran_COMPILER_ID MATCHES "GNU")

MOM6-interface/CMakeLists.txt

@@ -28,7 +28,7 @@ include("mom6_files.cmake")
 add_library(mom6_obj OBJECT ${mom6_src_files})
 set_target_properties(mom6_obj PROPERTIES Fortran_MODULE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/mod)
 target_include_directories(mom6_obj PRIVATE $<BUILD_INTERFACE:${CMAKE_CURRENT_BINARY_DIR}/mod>)
-target_include_directories(mom6_obj PRIVATE $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/MOM6/config_src/dynamic>
+target_include_directories(mom6_obj PRIVATE $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/MOM6/config_src/memory/dynamic_nonsymmetric>
                                             $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/MOM6/src/framework>)
 target_link_libraries(mom6_obj PRIVATE fms
                                        NetCDF::NetCDF_Fortran)
@@ -40,7 +40,7 @@ target_link_libraries(mom6_obj PRIVATE fms
 add_library(mom6_nuopc_obj OBJECT ${mom6_nuopc_src_files})
 set_target_properties(mom6_nuopc_obj PROPERTIES Fortran_MODULE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/mod)
 target_include_directories(mom6_nuopc_obj PRIVATE $<BUILD_INTERFACE:${CMAKE_CURRENT_BINARY_DIR}/mod>)
-target_include_directories(mom6_nuopc_obj PRIVATE $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/MOM6/config_src/dynamic>
+target_include_directories(mom6_nuopc_obj PRIVATE $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/MOM6/config_src/memory/dynamic_nonsymmetric>
                                                   $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/MOM6/src/framework>)
 target_link_libraries(mom6_nuopc_obj PRIVATE mom6_obj
                                              fms
@@ -55,7 +55,7 @@ add_library(mom6 STATIC $<TARGET_OBJECTS:mom6_obj>
                         $<TARGET_OBJECTS:mom6_nuopc_obj>)
 target_include_directories(mom6 PUBLIC $<BUILD_INTERFACE:${CMAKE_CURRENT_BINARY_DIR}/mod>
                                        $<INSTALL_INTERFACE:mod>)
-target_include_directories(mom6 PUBLIC $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/MOM6/config_src/dynamic>
+target_include_directories(mom6 PUBLIC $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/MOM6/config_src/memory/dynamic_nonsymmetric>
                                        $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/MOM6/src/framework>)
 target_link_libraries(mom6 PUBLIC fms
                                   esmf
@@ -73,7 +73,7 @@ if(MOM6SOLO)
   set_target_properties(mom6solo PROPERTIES Fortran_MODULE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/mod_solo)
   target_include_directories(mom6solo PRIVATE $<BUILD_INTERFACE:${CMAKE_CURRENT_BINARY_DIR}/mod>)
   target_include_directories(mom6solo PRIVATE $<BUILD_INTERFACE:${CMAKE_CURRENT_BINARY_DIR}/mod_solo>)
-  target_include_directories(mom6solo PRIVATE $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/MOM6/config_src/dynamic>
+  target_include_directories(mom6solo PRIVATE $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/MOM6/config_src/memory/dynamic_nonsymmetric>
                                               $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/MOM6/src/framework>)
   target_link_libraries(mom6solo PRIVATE mom6_obj
                                          fms

MOM6-interface/mom6_files.cmake

@@ -97,40 +97,28 @@ list(APPEND mom6_src_files
   MOM6/src/framework/MOM_array_transform.F90
   MOM6/src/framework/MOM_checksums.F90
   MOM6/src/framework/MOM_coms.F90
-  MOM6/src/framework/MOM_coms_infra.F90
-  MOM6/src/framework/MOM_constants.F90
   MOM6/src/framework/MOM_coupler_types.F90
-  MOM6/src/framework/MOM_couplertype_infra.F90
   MOM6/src/framework/MOM_cpu_clock.F90
-  MOM6/src/framework/MOM_cpu_clock_infra.F90
   MOM6/src/framework/MOM_data_override.F90
-  MOM6/src/framework/MOM_data_override_infra.F90
-  MOM6/src/framework/MOM_diag_manager_infra.F90
   MOM6/src/framework/MOM_diag_mediator.F90
   MOM6/src/framework/MOM_diag_remap.F90
   MOM6/src/framework/MOM_diag_vkernels.F90
   MOM6/src/framework/MOM_document.F90
-  MOM6/src/framework/MOM_domain_infra.F90
   MOM6/src/framework/MOM_domains.F90
   MOM6/src/framework/MOM_dyn_horgrid.F90
   MOM6/src/framework/MOM_ensemble_manager.F90
-  MOM6/src/framework/MOM_ensemble_manager_infra.F90
   MOM6/src/framework/MOM_error_handler.F90
-  MOM6/src/framework/MOM_error_infra.F90
   MOM6/src/framework/MOM_file_parser.F90
   MOM6/src/framework/MOM_get_input.F90
   MOM6/src/framework/MOM_hor_index.F90
   MOM6/src/framework/MOM_horizontal_regridding.F90
-  MOM6/src/framework/MOM_interp_infra.F90
   MOM6/src/framework/MOM_interpolate.F90
   MOM6/src/framework/MOM_intrinsic_functions.F90
   MOM6/src/framework/MOM_io.F90
-  MOM6/src/framework/MOM_io_infra.F90
   MOM6/src/framework/MOM_random.F90
   MOM6/src/framework/MOM_restart.F90
   MOM6/src/framework/MOM_safe_alloc.F90
   MOM6/src/framework/MOM_string_functions.F90
-  MOM6/src/framework/MOM_time_manager.F90
   MOM6/src/framework/MOM_unit_scaling.F90
   MOM6/src/framework/MOM_write_cputime.F90
 
@@ -265,21 +253,34 @@ list(APPEND mom6_src_files
   MOM6/config_src/external/ODA_hooks/ocean_da_types.F90
   MOM6/config_src/external/ODA_hooks/write_ocean_obs.F90
 
+  MOM6/config_src/infra/FMS1/MOM_coms_infra.F90
+  MOM6/config_src/infra/FMS1/MOM_constants.F90
+  MOM6/config_src/infra/FMS1/MOM_couplertype_infra.F90
+  MOM6/config_src/infra/FMS1/MOM_cpu_clock_infra.F90
+  MOM6/config_src/infra/FMS1/MOM_data_override_infra.F90
+  MOM6/config_src/infra/FMS1/MOM_diag_manager_infra.F90
+  MOM6/config_src/infra/FMS1/MOM_domain_infra.F90
+  MOM6/config_src/infra/FMS1/MOM_ensemble_manager_infra.F90
+  MOM6/config_src/infra/FMS1/MOM_error_infra.F90
+  MOM6/config_src/infra/FMS1/MOM_interp_infra.F90
+  MOM6/config_src/infra/FMS1/MOM_io_infra.F90
+  MOM6/config_src/infra/FMS1/MOM_time_manager.F90
+
 )
 
 list(APPEND mom6_nuopc_src_files
-  MOM6/config_src/nuopc_driver/mom_cap.F90
-  MOM6/config_src/nuopc_driver/time_utils.F90
-  MOM6/config_src/nuopc_driver/mom_cap_methods.F90
-  MOM6/config_src/nuopc_driver/mom_cap_time.F90
-  MOM6/config_src/nuopc_driver/mom_ocean_model_nuopc.F90
-  MOM6/config_src/nuopc_driver/mom_surface_forcing_nuopc.F90
+  MOM6/config_src/drivers/nuopc_cap/mom_cap.F90
+  MOM6/config_src/drivers/nuopc_cap/time_utils.F90
+  MOM6/config_src/drivers/nuopc_cap/mom_cap_methods.F90
+  MOM6/config_src/drivers/nuopc_cap/mom_cap_time.F90
+  MOM6/config_src/drivers/nuopc_cap/mom_ocean_model_nuopc.F90
+  MOM6/config_src/drivers/nuopc_cap/mom_surface_forcing_nuopc.F90
 )
 
 list(APPEND mom6_solo_src_files
-  MOM6/config_src/solo_driver/MESO_surface_forcing.F90
-  MOM6/config_src/solo_driver/MOM_driver.F90
-  MOM6/config_src/solo_driver/MOM_surface_forcing.F90
-  MOM6/config_src/solo_driver/atmos_ocean_fluxes.F90
-  MOM6/config_src/solo_driver/user_surface_forcing.F90
+  MOM6/config_src/drivers/solo_driver/MESO_surface_forcing.F90
+  MOM6/config_src/drivers/solo_driver/MOM_driver.F90
+  MOM6/config_src/drivers/solo_driver/MOM_surface_forcing.F90
+  MOM6/config_src/drivers/solo_driver/atmos_ocean_fluxes.F90
+  MOM6/config_src/drivers/solo_driver/user_surface_forcing.F90
 )

modulefiles/cheyenne.gnu/fv3

@@ -42,4 +42,4 @@ module load nemsio/2.5.2
 module load sp/2.3.3
 module load w3emc/2.7.3
 module load w3nco/2.4.1
-module load upp/10.0.0
+module load upp/10.0.4

modulefiles/cheyenne.gnu/fv3_debug

@@ -42,4 +42,4 @@ module load nemsio/2.5.2
 module load sp/2.3.3
 module load w3emc/2.7.3
 module load w3nco/2.4.1
-module load upp/10.0.0
+module load upp/10.0.4

modulefiles/cheyenne.intel/fv3

@@ -42,4 +42,4 @@ module load nemsio/2.5.2
 module load sp/2.3.3
 module load w3emc/2.7.3
 module load w3nco/2.4.1
-module load upp/10.0.0
+module load upp/10.0.4

modulefiles/cheyenne.intel/fv3_debug

@@ -42,4 +42,4 @@ module load nemsio/2.5.2
 module load sp/2.3.3
 module load w3emc/2.7.3
 module load w3nco/2.4.1
-module load upp/10.0.0
+module load upp/10.0.4