Skip to content

CAM Development Workflow in GitHub

Jump to bottom
goldy edited this page Jun 9, 2021 · 48 revisions

This page contains the standard workflow for development and/or bug fixes for the NCAR Community Atmosphere Model (CAM) using Github.

Contents

This page is a work in progress, and will likely be edited frequently over the next several weeks/months.

Please use any recommendations here with caution until the workflow is finalized and becomes "official".

How to notify NCAR of a problem, bug, or to request a new feature in CAM

  1. First, search the DiscussCESM forums to see if your problem has already been mentioned, and if a useful answer has been provided.

  2. If the problem or bug hasn't been noted on the forums, then create a new topic and describe the problem or bug there. If the problem has already been described in the forums, but hasn't received a useful answer, then please add a post to that topic indicating that you also ran into this particular problem. Both of these options will require you to either log-in to the forums, or to register if you don't already have an account.

  3. If the problem or bug is found to be in need of fixing, then a GitHub issue will be created and added to CAM's issue list, and will remain open and searchable until the problem or bug has been fixed by an NCAR AMP scientist or software engineer. If you already have the fix or solution on your own personal GitHub fork, then please notify the scientist or engineer so that your fix can be implemented into CAM itself (see "How to make/store revisions to your personal CAM repository" below).

If you are an AMP scientist or engineer who needs to create a CAM Github issue, then instructions for doing so can be found here.

How to make/store revisions to your personal CAM repository (GitHub fork)

  1. You will work on your fork in a clone. If you do not already have a fork of ESCOMP/CAM, create one. See Working with clones for an example of the button used to create a fork.

  2. If you do not already have a clone, create one:

% git clone https://github.com/<GitHub userid>/CAM
% cd CAM
% git remote add ESCOMP https://github.com/ESCOMP/CAM
% git fetch --tags ESCOMP

For help setting up a new clone see Working with clones.

  1. Create and checkout a branch for your work. Normally, your branch should start from the ESCOMP development branch:
% git branch <new_branch_name> ESCOMP/cam_development
% git checkout <new_branch_name>

unless you are specifically working on a release bug fix:

% git branch <new_branch_name> ESCOMP/cam_cesm2_1_rel
% git checkout <new_branch_name>

or are starting a bug fix from a specific CAM tag:

% git branch <new_branch_name> cam6_2_000
% git checkout <new_branch_name>

where <new_branch_name> should describe the change you are going to make (the second argument is where that branch begins).

  1. Make modifications or changes to the branch as you see fit and run some relevant tests.

  2. Commit new changes to local git clone, e.g.:

% git add <new_file(s)>
% git commit -am "<description of changes>"

where <description of changes> should describe all the changes in this commit. To enter a longer commit message, omit the m and the "<description of changes>" and an editor window should open that allows you to enter a detailed message. Note that the first command is only needed if you added new files to your clone (but always take a moment to think about it as forgotten files are a common source of problems that occur during testing). If you aren't sure what files have been added or removed, then type the command:

% git status

which will list all added, removed, and modified files, including files which are in the git repo's directories, but which are not being tracked by git itself.

  1. Push changes back to personal CAM Fork:
% git push <origin> <new_branch_name>

where <origin> is the name of the source repository (the default name is 'origin', you might have chosen a different name when you created the clone).

  1. Run CAM tests You should run tests fairly frequently as you develop new code or make modifications to existing code. In particular, it is important that there is a test for your simulation -- something that tests the new feature. All tests are run from the <root>/cime/scripts directory.
  • Run an individual test: To run an individual test, you need a test type (<TEST>) compset (<COMPSET>), a model grid (<GRID>), a machine (<MACHINE>) and a compiler (<COMPILER>). The create_test command will look like this:
% ./create_test <TEST>.<GRID>.<COMPSET>.<MACHINE>_<COMPILER>[.cam-<testmod>]

If you leave out the machine/compiler specification (<MACHINE>_<COMPILER>), CIME will choose a default compiler for the machine on which you are testing. However, you can only do this if you do not have any testmods.

A CAM test example is:

% ./create_test SMS_D_Ln9.f19_f19_mg17.FW4madSD.hobart_nag.cam-outfrq9s

where the last field is the particular configuration and run time modifications for this test. See the CIME manual section on testing for more details.

  • Run the aux_cam test suite: The aux_cam suite contains several tests which cover a variety of CAM simulation types. To run it, use this form of create_test:
% ./create_test --xml-category aux_cam --xml-compiler nag --test-id foo --output-root <scratch_dir> --test-root <scratch_dir>
  • To see the available tests:
% ./query_testlists

Use the --help option for details.

Finally, if running tests on Cheyenne, make sure to use the qcmd command to avoid running the test on the log-in nodes, like so:

% qcmd -- ./create_test <test_args>

where <test_args> are all the different create_test input arguments and flags described above.

Note that the master list of CAM tests is in <CAMROOT>/cime_config/testdefs/testlist_cam.xml and the testmods are in <CAMROOT>/cime_config/testdefs/testmods_dirs.

How to submit code changes to be included in ESCOMP/CAM

  1. Update the ChangeLog describing the code changes made, as well as the results of the CAM tests. The ChangeLog itself can be found in your local branch and/or fork here:
<CAM>/doc/ChangeLog

where <CAM> is the name of the local directory in which you have cloned your CAM fork. Once your ChangeLog additions have been made, commit and push the modified ChangeLog back to your fork using the same instructions above for the source code modifications. Please note that you do not need to run any additional tests, as long as the only file you changed is the ChangeLog itself.

  1. Make sure there is an open issue on ESCOMP/CAM describing the need for your change.

    • If you are a developer, follow the instructions for opening a new issue. If not, contact an AMP scientist or software engineer who can work with you to open an appropriate issue.
    • Before creating the Pull Request, be sure the issue is up to date.

  2. Be sure your code conforms to the CAM Coding Standards.

  3. Open a Pull Request (PR) or work with a CAM software engineer.

    • Update the issue and indicate that your code changes are ready to be added to CAM development.

    • Go to your CAM fork's GitHub page, and select the "Pull requests" tab:

    • Create a new pull request using the "new pull request" button:

    • Set the "base repository" to "ESCOMP/CAM", and the "base" to your branch of interest (likely "cam_development") -- IMPORTANT NOTE -- If you believe your comparison branch is not cam_development, please contact a CAM SE to verify that this is correct:

    • Set the "head repository" to your CAM fork repo, and the "compare" to your newly created branch:

    • If the phrase "Able to merge" is present, then select "create pull request":

    • Add a title for your request, along with a description of what your pull request is doing. Fill out all fields in the template while removing any template comments. Be sure to list the issue(s) which are addressed by this pull request, taking care to use the correct syntax.
    • Select the appropriate labels for this PR (e.g., "bug fix", "enhancement")
    • To see the process that happens after you issue the request you may follow your issue on the ESCOMP/CAM project board.
    • If you are asked to make changes to your PR:
      • Make the requested changes on your branch
      • Re-run the test(s) you used to verify your work
      • Commit the changes to your branch
      • Push the commit(s) back to your GitHub CAM fork

  4. If you are able to select a reviewer, please select the primary CAM software engineer (SE) who helped you during this process. This person will perform the preliminary review before putting it out for full code review. If there was no SE involved, do not select anyone and reviewers will be selected for you.

  5. Once your PR moves up the queue and is ready for code review, you will see additional reviewers added to your PR. Please note that requests for changes show up as conversations in your PR. The person making the request will hit the "Resolve Conversation" button when they are happy with your changes, so you should not close them.

  6. When you feel you have addressed all reviewer change requests, ask all reviewers to re-review the PR. To do this, click the "Re-request review" icon (two circular arrows) to the right of each reviewer's GitHub ID in the "Reviewers" list in the upper-right of the PR page. This notifies each reviewer to re-review the code. Repeat the last step and this step until there are no more change requests.

  7. Once the changes are complete, your PR will be put in line for an upcoming CAM tag, usually the next one. When that slot opens, the primary CAM SE will run the CAM regression tests, fix any issues which may be uncovered and make final edits to the ChangeLog. At this point you have two choices on how to bring back the changes made by the CAM SE to your fork.

    1. Change permissions in your CAM fork and allow the primary CAM SE write permission to your repo. That person will then push the changes back to your fork directly. You will be able to see the changes after this has been pushed back if you want.
    2. The CAM SE will open a Pull Request to your fork with the required changes, and you will review the changes and accept them back into your fork.

  8. The CAM SE will then hit the button on GitHub to bring your changes into the CAM GitHub repository.

  9. Finally, if you are an NCAR Software Engineer, and need to bring in code changes to an official CAM branch yourself, then you must follow the specialized CAM SE workflow.

How to remove old or unused branches

Once your modifications have been merged into the official CAM repo, you may have no more use for the local fork branch created to develop those modifications. In that case, you can remove the branch both from your local cloned repo and your GitHub CAM fork:

  1. First, make sure your local repo isn't checking out the old branch, by simply checking out a different branch:
    git checkout <some_other_branch>
  1. Then, remove branch from local repo:
    git branch -d <branch_name>
  1. Finally, remove branch from personal fork repo:
    git push --delete <origin> <branch_name>

You can also remove the branch via GitHub's user interface.

CAM wiki

CAM Documentation

CAM Model Development

Clone this wiki locally