Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(1) update MOM6 to 20230206 main branch (2) remove cpld_esmfthread_P8 (3) hafs bug fix #1606

Merged

Conversation

jiandewang
Copy link
Collaborator

@jiandewang jiandewang commented Feb 9, 2023

Description

MOM6 main branch is updated on 20230206, this is based on EMC ungridded dimension pstokes PR (mom-ocean/MOM6#1591), but being added through cherry-pick the ci failure fixing from dev/gfdl.
need to make corresponding updating in dev/emc and ufs-weather-model. No answer change is expected.

Top of commit queue on: TBD

Input data additions/changes

  • No changes are expected to input data.
  • There will be new input data.
  • Input data will be updated.

Anticipated changes to regression tests:

  • No changes are expected to any regression test.
  • Changes are expected to the following tests:
a new test is added, hafs_regional_storm_following_1nest_atm_ocn_debug

Subcomponents involved:

  • AQM
  • CDEPS
  • CICE
  • CMEPS
  • CMakeModules
  • FV3
  • GOCART
  • HYCOM
  • MOM6
  • NOAHMP
  • WW3
  • stochastic_physics
  • none

Combined with PR's (If Applicable):

Commit Queue Checklist:

  • Link PR's from all sub-components involved
  • Confirm reviews completed in sub-component PR's
  • Add all appropriate labels to this PR.
  • Run full RT suite on either Hera/Cheyenne with both Intel/GNU compilers
  • Add list of any failed regression tests to "Anticipated changes to regression tests" section.

Linked PR's and Issues:

Testing Day Checklist:

  • This PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR.
  • Move new/updated input data on RDHPCS Hera and propagate input data changes to all supported systems.

Testing Log (for CM's):

  • RDHPCS
    • Intel
      • Hera
      • Orion
      • Jet
      • Gaea
      • Cheyenne
    • GNU
      • Hera
      • Cheyenne
  • WCOSS2
    • Dogwood/Cactus
    • Acorn
  • CI
    • Completed
  • opnReqTest

@jiandewang
Copy link
Collaborator Author

HERA_intel-log.pdf

@jkbk2004 jkbk2004 added Baseline Updates Current baselines will be updated. Waiting for Reviews The PR is waiting for reviews from associated component PR's. Ready for Commit Queue The PR is ready for the Commit Queue. All checkboxes in PR template have been checked. labels Feb 9, 2023
@DeniseWorthen
Copy link
Collaborator

Do we need a new baseline? Can't we just add the one new debug test to the existing baseline?

@jkbk2004
Copy link
Collaborator

jkbk2004 commented Feb 9, 2023

Do we need a new baseline? Can't we just add the one new debug test to the existing baseline?

Sure, I can revert the BL_DATE change and do manually add hafs_regional_storm_following_1nest_atm_ocn_debug.

@jkbk2004
Copy link
Collaborator

jkbk2004 commented Feb 9, 2023

@BrianCurtis-NOAA I am going to manually create hafs_regional_storm_following_1nest_atm_ocn_debug and copy over to develop-20230201

@BrianCurtis-NOAA
Copy link
Collaborator

@jkbk2004 Can you explain how the baselines are identical? Debug should run optimizations different than non-debug which would cause different results.

@jkbk2004
Copy link
Collaborator

@jkbk2004 Can you explain how the baselines are identical? Debug should run optimizations different than non-debug which would cause different results.

I think there is no rounding-off difference with -g -traceback or debug options at lower level optimization.

@jkbk2004
Copy link
Collaborator

I agree we should see some level of rounding-off difference by triggering different level of optimization. The difference not necessarily by debug option. I think this is another reason to double check our compile options.

@DeniseWorthen
Copy link
Collaborator

I still think we need to be sure about the compile, it makes no sense to me that the answers aren't different. For example, the atmf000.nc control_p8 and control_debug_p8 files are different. I think that is after a single startup timestep (ie, 000 is really after the first timestep of the atm).

@junwang-noaa
Copy link
Collaborator

@jkbk2004 As I explained in the slack, the results from the new HAFS debug test are different from the HAFS control hafs_regional_storm_following_1nest_atm_ocn results. The baseline filenames are different, and they were put in the control baseline by mistake, but the RT will still pass because the code changes have no impact on other tests and previous baseline is not touched in the full RT. Hope this clarifies the confusion.

@DeniseWorthen
Copy link
Collaborator

I know what is happening now. The 001 files from the debug run are still present on hera in the non-debug baseline hafs_regional_storm_following_1nest_atm_ocn.

We should remove the files that were created by the debug test from the non-debug baseline.

@BrianCurtis-NOAA
Copy link
Collaborator

I know what is happening now. The 001 files from the debug run are still present on hera in the non-debug baseline hafs_regional_storm_following_1nest_atm_ocn.

We should remove the files that were created by the debug test from the non-debug baseline.

The next PR will create new baselines, right? We can just wait for that, it won't hurt to keep the extra baselines in the non-debug directory.

@DeniseWorthen
Copy link
Collaborator

@BrianCurtis-NOAA Normally I might agree about the next baseline, but in this case I think the amount of confusion that has been generated makes it worthwhile to clean up.

@BrianCurtis-NOAA
Copy link
Collaborator

@BrianCurtis-NOAA Normally I might agree about the next baseline, but in this case I think the amount of confusion that has been generated makes it worthwhile to clean up.

Dogwood is cleaned up. New debug baselines are created in the correct dir. Dogwood is set for this merge.

Acorn can officially be skipped. The RT's will compile but not run, no logs are output. Seems a system issue, but it will need attention ASAP for the next set of PR's

@jkbk2004
Copy link
Collaborator

OK! hafs_regional_storm_following_1nest_atm_ocn and hafs_regional_storm_following_1nest_atm_ocn_debug are all updated ok on hera/orion/cheyenne. gaea sis down today. I will catch up gaea later. We can move to start merging process.

@BrianCurtis-NOAA
Copy link
Collaborator

the ecflow nodes on Dogwood went down before i could re-run one test that failed, but it passed earlier. The debug and non-debug tests passed, so WCOSS2 is good to go.

@jkbk2004
Copy link
Collaborator

cool! I am merging NOAA-EMC/fv3atm#622. @jiandewang can you merging mom6 pr?

@jkbk2004
Copy link
Collaborator

@jiandewang NOAA-EMC/fv3atm#622 was merged. Please, go ahead to update submodule pointers and revert the changes in gitmodules after you merge mom6 pr.

@jiandewang
Copy link
Collaborator Author

@jiandewang NOAA-EMC/fv3atm#622 was merged. Please, go ahead to update submodule pointers and revert the changes in gitmodules after you merge mom6 pr.

MOM6 merged

@jkbk2004
Copy link
Collaborator

@jiandewang cool! yeah, we can merge once submodule pointers (fv3/mom6) are updated and gitmodules changes are reverted.

@DeniseWorthen
Copy link
Collaborator

@jiandewang The hash for MOM6 needs to be d0c4550

@jiandewang
Copy link
Collaborator Author

I reverted MOM6 and committed, but for FV3, this is what I did (after revert its submodule):
cd FV3
git fetch
git checkout develop

but I got many files being removing information, am I doing correct ? (I haven't committed yet)
/scratch1/NCEPDEV/climate/Jiande.Wang/MOM6-update/MOM6-update-20230206/Final/ufs-weather-model/FV3[150]git checkout develop
warning: unable to rmdir 'upp': Directory not empty
M atmos_cubed_sphere
M ccpp/framework
M ccpp/physics
Previous HEAD position was ffeb762 Merge remote-tracking branch 'upstream/develop' into hafs_movingnest
Switched to branch 'develop'
Your branch and 'origin/develop' have diverged,
and have 1 and 273 different commits each, respectively.
(use "git pull" to merge the remote branch into yours)
/scratch1/NCEPDEV/climate/Jiande.Wang/MOM6-update/MOM6-update-20230206/Final/ufs-weather-model/FV3[151]git pull
Removing time_utils.F90
Removing tests/tests/fv3_stretched_nest
Removing tests/tests/fv3_stretched
Removing tests/tests/fv3_restart
Removing tests/tests/fv3_read_inc
Removing tests/tests/fv3_gfdlmp
Removing tests/tests/fv3_decomp
Removing tests/tests/fv3_control
Removing tests/tests/fv3_2threads
Removing tests/tail.h
Removing tests/run_test.sh
Removing tests/rt_utils.sh
Removing tests/rt_fv3.sh
Removing tests/rt.sh
Removing tests/rt.conf
Removing tests/head.h
Removing tests/fv3_conf/stretched-nest-input.nml.IN
Removing tests/fv3_conf/stretched-input.nml.IN
Removing tests/fv3_conf/input_nest02.nml.IN
Removing tests/fv3_conf/input.nml.IN
Removing tests/fv3_conf/gfdlmp_run.IN
Removing tests/fv3_conf/gfdlmp.nml.IN
Removing tests/fv3_conf/fv3_stretched_run.IN
Removing tests/fv3_conf/fv3_run.IN
Removing tests/fv3_conf/fv3_qsub.IN_theia
Removing tests/fv3_conf/fv3_bsub.IN_wcoss_cray
Removing tests/fv3_conf/fv3_bsub.IN
Removing tests/detect_machine.sh
Removing tests/default_vars.sh
Removing tests/compile.sh
Removing tests/atparse.bash
Removing tests/abort_dep_tasks.py
Removing tests/RegressionTests_wcoss_cray.log
Removing tests/RegressionTests_theia.log
Removing tests/Compile_wcoss_cray.log
Removing tests/Compile_theia.log
Removing namphysics/physics/shalcnv.f
Removing namphysics/physics/sascnvn.f
Removing namphysics/physics/radsw_param.f
Removing namphysics/physics/radsw_main_nmmb.f
Removing namphysics/physics/radsw_datatb.f
Removing namphysics/physics/radlw_param.f
Removing namphysics/physics/radlw_main_nmmb.f
Removing namphysics/physics/radlw_datatb.f
Removing namphysics/physics/radiation_surface_nmmb.f
Removing namphysics/physics/radiation_gases_nmmb.f
Removing namphysics/physics/radiation_clouds_nmmb.f
Removing namphysics/physics/radiation_astronomy_nmmb.f
Removing namphysics/physics/radiation_aerosols_nmmb.f
Removing namphysics/physics/rad_initialize_nmmb.f
Removing namphysics/physics/precpd_nmmb.f
Removing namphysics/physics/physparam.f
Removing namphysics/physics/physcons.f90
Removing namphysics/physics/ozne_def.f
Removing namphysics/physics/noahmp_tables.f90
Removing namphysics/physics/namelist_soilveg.f
Removing namphysics/physics/moninq.f
Removing namphysics/physics/moninedmf.f
Removing namphysics/physics/module_mp_thompson.F90
Removing namphysics/physics/module_mp_radar.F90
Removing namphysics/physics/module_TURBULENCE.F90
Removing namphysics/physics/module_SF_URBAN.F90
Removing namphysics/physics/module_SF_JSFC.F90
Removing namphysics/physics/module_SF_GFDL.F90
Removing namphysics/physics/module_RA_RRTM.F90
Removing namphysics/physics/module_RA_GFDL.F90
Removing namphysics/physics/module_RADIATION.F90
Removing namphysics/physics/module_MP_WSM6.F90
Removing namphysics/physics/module_MP_GFS.F90
Removing namphysics/physics/module_MP_FER_HIRES.F90
Removing namphysics/physics/module_MP_ETANEW.F90
Removing namphysics/physics/module_MICROPHYSICS.F90
Removing namphysics/physics/module_LS_NOAHLSM.F90
Removing namphysics/physics/module_LS_LISS.F90
Removing namphysics/physics/module_KINDS.F90
Removing namphysics/physics/module_GWD.F90
Removing namphysics/physics/module_DIAGNOSE.F90
Removing namphysics/physics/module_CU_SCALE.F90
Removing namphysics/physics/module_CU_SASHUR.F90
Removing namphysics/physics/module_CU_SAS.F90
Removing namphysics/physics/module_CU_BMJ.F90
Removing namphysics/physics/module_CONVECTION.F90
Removing namphysics/physics/module_CONTROL.F90
Removing namphysics/physics/module_CONSTANTS.F90
Removing namphysics/physics/module_BL_MYJPBL.F90
Removing namphysics/physics/module_BL_GFSPBLHUR.F90
Removing namphysics/physics/module_BL_GFSPBLEDMFHUR.F90
Removing namphysics/physics/module_BL_GFSPBL.F90
Removing namphysics/physics/mfshalcnv.f
Removing namphysics/physics/mfpbl.f
Removing namphysics/physics/mfdeepcnv.f
Removing namphysics/physics/mersenne_twister.f
Removing namphysics/physics/machine.F
Removing namphysics/physics/iounitdef.f
Removing namphysics/physics/h2o_def.f
Removing namphysics/physics/gscond.f
Removing namphysics/physics/grrad_nmmb.f
Removing namphysics/physics/gfdl_cloud_microphys.F90
Removing namphysics/physics/funcphys.f90
Removing namphysics/physics/GFDL_parse_tracers.F90
Removing namphysics/makefile
Removing namphysics/NAM_layer/module_SOLVER_GRID_COMP.F90
Removing namphysics/NAM_layer/NAM_typedefs.F90
Removing namphysics/NAM_layer/NAM_restart.F90
Removing namphysics/NAM_layer/NAM_radiation_driver.F90
Removing namphysics/NAM_layer/NAM_driver.F90
Removing namphysics/NAM_layer/NAM_diagnostics.F90
Removing namphysics/NAM_layer/NAM_abstraction_layer.F90
Removing mkDepends.pl
Removing makefile
Removing ipd/makefile
Removing ipd/IPD_typedefs.F90
Removing ipd/IPD_driver.F90
Removing io/post_gfs_stub.F90
Removing io/post_gfs.F90
Removing io/module_write_nemsio.F90
Removing io/makefile
Removing io/ffsync.F90
Removing gfsphysics/physics/wv_saturation.F
Removing gfsphysics/physics/wam_f107_kp_mod.f90
Removing gfsphysics/physics/ugwp_driver_v0.f
Removing gfsphysics/physics/tridi2t3.f
Removing gfsphysics/physics/tracer_const_h.f
Removing gfsphysics/physics/surface_perturbation.F90
Removing gfsphysics/physics/shalcv_opr.f
Removing gfsphysics/physics/shalcv_fixdp.f
Removing gfsphysics/physics/shalcv_1lyr.f
Removing gfsphysics/physics/shalcv.f
Removing gfsphysics/physics/shalcnv.f
Removing gfsphysics/physics/sflx.f
Removing gfsphysics/physics/sfcsub.F
Removing gfsphysics/physics/sfc_sice.f
Removing gfsphysics/physics/sfc_ocean.f
Removing gfsphysics/physics/sfc_nst.f
Removing gfsphysics/physics/sfc_noahmp_drv.f
Removing gfsphysics/physics/sfc_drv.f
Removing gfsphysics/physics/sfc_diff.f
Removing gfsphysics/physics/sfc_diag.f
Removing gfsphysics/physics/sfc_cice.f
Removing gfsphysics/physics/set_soilveg.f
Removing gfsphysics/physics/satmedmfvdifq.f
Removing gfsphysics/physics/satmedmfvdif.f
Removing gfsphysics/physics/sascnvn.f
Removing gfsphysics/physics/sascnv.f
Removing gfsphysics/physics/samfshalcnv.f
Removing gfsphysics/physics/samfdeepcnv.f
Removing gfsphysics/physics/samfaerosols.f
Removing gfsphysics/physics/rayleigh_damp_mesopause.f
Removing gfsphysics/physics/rayleigh_damp.f
Removing gfsphysics/physics/rascnvv2.f
Removing gfsphysics/physics/radsw_param.f
Removing gfsphysics/physics/radsw_main.f
Removing gfsphysics/physics/radsw_datatb.f
Removing gfsphysics/physics/radlw_param.f

@BrianCurtis-NOAA
Copy link
Collaborator

BrianCurtis-NOAA commented Feb 14, 2023

@jiandewang I'm going to guess you checked out your own develop branch and not the one from FV3? you need to add https://github.com/NOAA-EMC/fv3atm to your remotes, update it, then git checkout <NOAA-EMC/fv3atm remote name> develop

i.e. in FV3:

git remote add upstream https://github.com/NOAA-EMC/fv3atm
git remote update
git checkout upstream develop (or maybe upstream/develop)
cd ..
git add FV3
git commit -m "Update FV3 hash"
git push origin update-MOM6-20230206

@junwang-noaa
Copy link
Collaborator

junwang-noaa commented Feb 14, 2023 via email

@jiandewang
Copy link
Collaborator Author

many thanks to Denise's help, FV3 and MOM6 are pointing to the right hash # now

@jkbk2004 jkbk2004 merged commit 16277fa into ufs-community:develop Feb 14, 2023
@jkbk2004 jkbk2004 mentioned this pull request Feb 14, 2023
38 tasks
@jiandewang jiandewang deleted the feature/update-MOM6-20230206 branch July 22, 2024 03:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Baseline Updates Current baselines will be updated. jenkins-ci Jenkins CI: ORT build/test on docker container Ready for Commit Queue The PR is ready for the Commit Queue. All checkboxes in PR template have been checked. Waiting for Reviews The PR is waiting for reviews from associated component PR's.
Projects
None yet
7 participants