Skip to content

Building and Using UFS GOCART on Orion

Jump to bottom
Ben Auer edited this page Mar 9, 2021 · 2 revisions

Building UFS-GOCART on Orion

This is a complete guide to for building GOCART on Mississippi State HPC2's Orion Computer. This guide assumes one can log into Orion and that they are running the Bash shell.

Note that these directions are subject to change as UFS-GOCART is constantly being updated. Please ask @rmontouro for assistance if you run into issues.

Obtaining UFS-GOCART

In your work directory, I suggest using:

/work/noaa/nems/<username>

Create a folder to house your build and associated experiments, then change directories to inside that folder. Next clone the development branch for UFS-GOCART using:

module load git
git clone -b feature/ufs_develop --recursive https://github.com/NOAA-EMC/FV3-GOCART

This should clone all the necessary repositories to get started working with UFS-GOCART.

Building UFS-GOCART

Assuming your still in your build and experiment directory, you can prepare your environment to build using the following commands:

cd FV3-GOCART
module use modulefiles/orion.intel
module load fv3

This will load all the necessary module files on Orion to provide all the dependencies for UFS-GOCART. Then one can build/re-build UFS-GOCART using:

./build.sh

Building and Using MAPL for UFS-GOCART

First, prepare your environment as you did for building UFS-GOCART, then you need to unload the MAPL environment using:

module unload mapl

Obtaining MAPL

Make sure you are in the build and experiment directory. You can now clone MAPL using:

git clone -b develop https://github.com/GEOS-ESM/MAPL.git

Note that UFS-GOCART currently requires you to use at least the develop branch of MAPL.

Building MAPL

Now we can build MAPL by doing the following:

cd MAPL
mkdir build
cd build
cmake .. -DCMAKE_INSTALL_PREFIX=../install \
         -DCMAKE_MODULE_PATH=$CMAKE_MODULE_PATH \ 
         -DCMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH \ 
         -DBUILD_WITH_FLAP=NO \
         -DCMAKE_BUILD_TYPE=Debug
gmake -j install

Note that you can switch Debug to Release if you want to turn off debugging flags in MAPL.

Creating a MAPL module file

Now you need to prepare a module file to load your own MAPL version of MAPL for use by UFS-GOCART. I suggest storing all your MAPL module files in:

/work/noaa/nems/<username>/modulefiles/mapl

Now change directories to where you want your MAPL module files to be stored and then create a new module file using the following template:

help([[
]])

local pkgName    = myModuleName()
local pkgVersion = myModuleVersion()
local pkgNameVer = myModuleFullName()

local hierA        = hierarchyA(pkgNameVer,2)
local mpiNameVer   = hierA[1]
local compNameVer  = hierA[2]
local mpiNameVerD  = mpiNameVer:gsub("/","-")
local compNameVerD = compNameVer:gsub("/","-")

conflict(pkgName)

local opt = os.getenv("HPC_OPT") or os.getenv("OPT") or "/opt/modules"

local base = "/path/to/MAPL/install"

prepend_path("CMAKE_MODULE_PATH", pathJoin(base, "share/MAPL/cmake"), ";")
prepend_path("CMAKE_PREFIX_PATH", base, ";")

setenv("MAPL_ROOT", base)
setenv("MAPL_INCLUDES", pathJoin(base,"include"))
setenv("MAPL_LIBS", pathJoin(base,"lib"))
setenv("MAPL_VERSION", pkgVersion)

whatis("Name: ".. pkgName)
whatis("Version: " .. pkgVersion)
whatis("Category: library")
whatis("Description: NASA MAPL library")

The only thing that needs to be changed is the base variable, and it needs to be set to the install directory you selected when building your version of MAPL.

Using your version of MAPL

Now you need to load your version of MAPL along side of the of the fv3 modules (except the unloaded mapl module). To do this run

module use /path/to/your/mapl/module/directory
module load mapl/<name of your mapl module>

Once you have loaded your version MAPL switch directories to your FV3-GOCART directory (where you cloned UFS-GOCART), and (re)-run

./build.sh

This will (re)build UFS-GOCART using the version MAPL that you just built.

Using UFS-GOCART

Change directories to your build and experiment directory, then create a new directory to contain your experiment. Switch to that directory and then copy the template experiment using:

cp -r  /work/noaa/stmp/rmontuor/nasa/exp/debug/template/run_c96 rundir

Now edit the file rundir/jobcard to set ufs_gocart_dir is an absolute path to your FV3-GOCART repository (where you built UFS-GOCART). Finally to run an experiment you do:

cd rundir
sbatch ./job_card

To check that your experiment ran successfully inspect the out file that was created for the following near its end:

  0:      PROGRAM nems      HAS ENDED.

  0: * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . 

  0: *****************RESOURCE STATISTICS*******************************

  0: The total amount of wall time                        = 290.473444

  0: The total amount of time in user mode                = 267.228628

  0: The total amount of time in sys mode                 = 6.737510

  0: The maximum resident set size (KB)                   = 882352

  0: Number of page faults without I/O activity           = 685990

  0: Number of page faults with I/O activity              = 52

  0: Number of times filesystem performed INPUT           = 119744

  0: Number of times filesystem performed OUTPUT          = 1782336

  0: Number of Voluntary Context Switches                 = 28262

  0: Number of InVoluntary Context Switches               = 236

  0: *****************END OF RESOURCE STATISTICS*************************

If this appears, then the UFS-GOCART experiment ran to completion. Note that all standard output is sent to out while all standard error output is sent to err. Also note that err maybe non-empty due to issues with the post-experiment finalization.

Clone this wiki locally