Skip to content

CAM Development Workflow in GitHub

Jump to bottom
cacraigucar edited this page Oct 6, 2020 · 48 revisions

This page contains the standard workflow for development and/or bug fixes for the NCAR Community Atmosphere Model (CAM) using Github.

Contents

This page is a work in progress, and will likely be edited frequently over the next several weeks/months.

Please use any recommendations here with caution until the workflow is finalized and becomes "official".

How to notify NCAR of a problem or bug in CAM

  1. First, search the DiscussCESM forums to see if your problem has already been mentioned, and if a useful answer has been provided.

  2. If the problem or bug hasn't been noted on the forums, then create a new topic and describe the problem or bug there. If the problem has already been described in the forums, but hasn't received a useful answer, then please add a post to that topic indicating that you also ran into this particular problem. Both of these options will require you to either log-in to the forums, or to register if you don't already have an account.

  3. If the problem or bug is found to be in need of fixing, then a GitHub issue will be created and added to CAM's issue list, and will remain open and searchable until the problem or bug has been fixed by an NCAR AMP scientist or software engineer. If you already have the fix or solution on your own personal GitHub fork, then please notify the scientist or engineer so that your fix can be implemented into CAM itself (see "How to make/store revisions to your personal CAM repository" below).

If you are an AMP scientist or engineer who needs to create a CAM Github issue, then instructions for doing so can be found here.

How to make/store revisions to your personal CAM repository (GitHub fork)

  1. You will work on your fork in a clone. If you do not already have a fork of ESCOMP/CAM, create one. See Working with clones for an example of the button used to create a fork.

  2. If you do not already have a clone, create one:

% git clone https://github.com/<GitHub userid>/CAM
% cd CAM
% git remote add ESCOMP https://github.com/ESCOMP/CAM
% git fetch --tags ESCOMP

For help setting up a new clone see Working with clones.

  1. Create and checkout a branch for your work. Normally, your branch should start from the ESCOMP development branch:
% git branch <new_branch_name> ESCOMP/cam_development
% git checkout <new_branch_name>

unless you are specifically working on a release bug fix:

% git branch <new_branch_name> ESCOMP/cam_cesm2_1_rel
% git checkout <new_branch_name>

or are starting a bug fix from a specific CAM tag:

% git branch <new_branch_name> cam6_2_000
% git checkout <new_branch_name>

where <new_branch_name> should describe the change you are going to make (the second argument is where that branch begins).

  1. Make modifications or changes to the branch as you see fit and run some relevant tests.

  2. Commit new changes to local git clone, e.g.:

% git add <new_file(s)>
% git commit -am "<description of changes>"

where <description of changes> should describe all the changes in this commit. To enter a longer commit message, omit the m and the "<description of changes>" and an editor window should open that allows you to enter a detailed message. Note that the first command is only needed if you added new files to your clone (but always take a moment to think about it as forgotten files are a common source of problems that occur during testing). If you aren't sure what files have been added or removed, then type the command:

% git status

which will list all added, removed, and modified files, including files which are in the git repo's directories, but which are not being tracked by git itself.

  1. Push changes back to personal CAM Fork:
% git push <origin> <new_branch_name>

where <origin> is the name of the source repository (the default name is 'origin', you might have chosen a different name when you created the clone).

  1. Run CAM tests You should run tests fairly frequently as you develop new code or make modifications to existing code. In particular, it is important that there is a test for your simulation -- something that tests the new feature. All tests are run from the <root>/cime/scripts directory.
  • Run an individual test: To run an individual test, you need a test type (<TEST>) compset (<COMPSET>), a model grid (<GRID>), a machine (<MACHINE>) and a compiler (<COMPILER>). The create_test command will look like this:
% ./create_test <TEST>.<GRID>.<COMPSET>.<MACHINE>_<COMPILER>

If you leave out the last part (<MACHINE>_<COMPILER>), CIME will choose a default compiler for the machine on which you are testing.

A CAM test example is:

% ./create_test SMS_D_Ln9.f19_f19_mg17.FW4madSD.hobart_nag.cam-outfrq9s

where the last field is the particular configuration and run time modifications for this test. See the CIME manual section on testing for more details.

  • Run the aux_cam test suite: The aux_cam suite contains several tests which cover a variety of CAM simulation types. To run it, use this form of create_test:
% ./create_test --xml-category aux_cam --xml-compiler nag --test-id foo --output-root <scratch_dir> --test-root <scratch_dir>
  • To see the available tests:
% ./query_testlists

Use the --help option for details.

Finally, if running tests on Cheyenne, make sure to use the qcmd command to avoid running the test on the log-in nodes, like so:

% qcmd -- ./create_test <test_args>

where <test_args> are all the different create_test input arguments and flags described above.

  1. Update the ChangeLog describing the code changes made, as well as the results of the CAM tests. The ChangeLog itself can be found in your local branch and/or fork here:
<CAM>/doc/ChangeLog

where <CAM> is the name of the local directory in which you have cloned your CAM fork. Once your ChangeLog additions have been made, commit and push the modified ChangeLog back to your fork using the same instructions above for the source code modifications. Please note that you do not need to run any additional tests, as long as the only file you changed is the ChangeLog itself.

  1. If you have never submitted a PR to ESCOMP/CAM, contact an AMP scientist or software engineer so we can add you to the list of developers. This will allow you can open a PR (and designate the assignee as described in step 10). If an issue does not exist, we can also help you open an appropriate issue.

  2. If you will submit your change as an update to ESCOMP/CAM, be sure an issue is open which describes the need for your change.

    • Update the issue and indicate that your code changes are ready to be added to CAM development.
    • On the top of the far right box on GitHub, is an assignee. Add the CAM SE in charge of scheduling (currently Cheryl Craig - github ID cacraigucar) to the issue. This step is very important as it will be the hand-off of your issue to one of the CAM SEs for it to be added to the integration schedule.
    • Go to your CAM fork's GitHub page, and select the "Pull requests" tab:

    • Create a new pull request using the "new pull request" button:

    • Set the "base repository" to "ESCOMP/CAM", and the "base" to your branch of interest (likely "cam_development")
      -- IMPORTANT NOTE -- If you believe your comparison branch is not cam_development, please contact a CAM SE to verify that this is correct:

    • Set the "head repository" to your CAM fork repo, and the "compare" to your newly created branch:

    • If the phrase "Able to merge" is present, then select "create pull request":

    • Add a title for your request, along with a description of what your pull request is doing. Fill out all fields in the template while removing any template comments. Be sure to list the issue(s) which are addressed by this pull request, taking care to use the correct syntax.
    • To see the process that happens after you issue the request you may follow your issue on the ESCOMP/CAM project board.
    • If you are asked to make changes to your PR:
      • Make the requested changes on your branch
      • Re-run the test(s) you used to verify your work
      • Commit the changes to your branch
      • Push the commit(s) back to your GitHub CAM fork

  3. If you are able to select a reviewer, please select the primary CAM software engineer who helped you during this process. This person will perform the preliminary review before putting it out for full code review. If there was no SE involved, please assign to cacraigucar and she will make the assignment for you. Note that later in the process you will see additional reviewers added to your PR.

  4. Finally, if you are an NCAR Software Engineer (SE), and need to bring in code changes to an official CAM branch yourself, then you must follow the specialized CAM SE workflow.

How to remove old or unused branches

Once your modifications have been merged into the official CAM repo, you may have no more use for the local fork branch created to develop those modifications. In that case, you can remove the branch both from your local cloned repo and your GitHub CAM fork:

  1. First, make sure your local repo isn't checking out the old branch, by simply checking out a different branch:
    git checkout <some_other_branch>
  1. Then, remove branch from local repo:
    git branch -d <branch_name>
  1. Finally, remove branch from personal fork repo:
    git push --delete <origin> <branch_name>

You can also remove the branch via GitHub's user interface.

CAM wiki

CAM Documentation

CAM Model Development

Clone this wiki locally