Transition CCPP to ufs-s2s-model (contains #7) #64

climbfuji · 2020-03-27T15:00:17Z

This PR contains and expands upon the original PR from @panll #7.

add CCPP to ufs-s2s-model
update submodule pointer for fv3atm to point to latest version for b4b identical results for the coupled model configuration (see PR Regain bit-for-bit identical results between IPD and CCPP for coupled model runs NOAA-EMC/fv3atm#86 for details)
add IPD REPRO, CCPP REPRO, CCPP PROD regression tests to NEMSCompsetRun
try to be (more) consistent by placing the mediator restart files of the warm start runs in the RESTART directory (old tests previously put them in MEDIATOR_after_Xday, whereas the new benchmark runs put them in RESTART)
update existing namelist templates (not sure if they are used at all; the standard control and gfdlmp templates) so that IPD and CCPP match each other as much as possible and that they include the latest development in fv3atm
temporary Python script to facilitate the process of creating baselines (to be abandoned as soon as possible when the entire process is revamped)

update

compsets/all_ccpp.input compsets/cpld_fv3_ccpp_mom6_cice_2d_atm_flux.input compsets/cpld_fv3_ccpp_mom6_cice_cold_atm_flux.input compsets/fv3mom6cice5_ccpp.input coupledFV3_CCPP_MOM6_CICE.appBuilder

… develop

…nd testing

…of regression test baseline

…rectories iteratively from rtgen.PID directories

… (Regain bit-for-bit identical results between IPD and CCPP for coupled model runs)

JessicaMeixner-NOAA · 2020-03-30T14:59:56Z

compsets/fv3mom6cice5.input

@@ -61,7 +61,7 @@ build fv3_mom6_cice.exe {

    # The NEMSAppBuilder is another embedded bash script in nemscommon.input 
    build=NEMSAppBuilder(NEMS.x="@[target]",modules.nems="@[modules.nems]", 
-                         OPTS="app=coupledFV3_MOM6_CICE",md5sum="@[md5sum]") 
+                         OPTS="app=coupledFV3_MOM6_CICE rebuild",md5sum="@[md5sum]") 


Why is "rebuild" added here? In https://github.com/ufs-community/ufs-s2s-model/blob/develop/compsets/nemscommon.input#L34 we have that all builds have to "rebuild"? This was added in the last commit because when adding a build with WW3, it was required that we rebuild always or we were having issues.

Oops. I think this was by coincidence after creating the CCPP versions and then trying to make them look the same. I can revert this change (and make the same change on the CCPP side).

It seems that the "rebuild" can then be removed from the other build commands as well, or, given that it is in nemscommon.input?

junwang-noaa · 2020-03-30T15:10:43Z

Jessica, Does this mean we have to have a clean build every time when we build the model? This could cause lots of inconvenience for debugging. So if this is the case, can we have a separate build for fv3-mom6-cice5-wave for 4 components test and an fv3-mom6-cice5 executable for 3 component tests until the issue with rebuild fv3-mom6-cice5-wave is resolved?

…

On Mon, Mar 30, 2020 at 11:03 AM Dom Heinzeller ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In compsets/fv3mom6cice5.input <#64 (comment)> : > @@ -61,7 +61,7 @@ build fv3_mom6_cice.exe { # The NEMSAppBuilder is another embedded bash script in nemscommon.input ***@***.******@***.***", - ***@***.***") + OPTS="app=coupledFV3_MOM6_CICE ***@***.***") It seems that the "rebuild" can then be removed from the other build commands as well, or, given that it is in nemscommon.input? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#64 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI7D6TPLTH2B446X32OCXZ3RKCYETANCNFSM4LVCESMQ> .

…build' in NEMSAppBuilder commands

climbfuji · 2020-03-30T15:12:28Z

Jessica, Does this mean we have to have a clean build every time when we build the model? This could cause lots of inconvenience for debugging. So if this is the case, can we have a separate build for fv3-mom6-cice5-wave for 4 components test and an fv3-mom6-cice5 executable for 3 component tests until the issue with rebuild fv3-mom6-cice5-wave is resolved?
…

Commit d179cc3 updates the baseline on hera (revert to new official baseline dir) and removes the redundant "rebuild" flags in the individual app builder sections.

SMoorthi-emc · 2020-03-30T15:28:15Z

Using compile.sh is way simpler!

…

On Mon, Mar 30, 2020 at 11:11 AM Jun Wang ***@***.***> wrote: Jessica, Does this mean we have to have a clean build every time when we build the model? This could cause lots of inconvenience for debugging. So if this is the case, can we have a separate build for fv3-mom6-cice5-wave for 4 components test and an fv3-mom6-cice5 executable for 3 component tests until the issue with rebuild fv3-mom6-cice5-wave is resolved? On Mon, Mar 30, 2020 at 11:03 AM Dom Heinzeller ***@***.***> wrote: > ***@***.**** commented on this pull request. > ------------------------------ > > In compsets/fv3mom6cice5.input > < #64 (comment) > > : > > > @@ -61,7 +61,7 @@ build fv3_mom6_cice.exe { > > # The NEMSAppBuilder is another embedded bash script in nemscommon.input > ***@***.******@***.***", > - ***@***.***") > + OPTS="app=coupledFV3_MOM6_CICE ***@***.***") > > It seems that the "rebuild" can then be removed from the other build > commands as well, or, given that it is in nemscommon.input? > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > < #64 (comment) >, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/AI7D6TPLTH2B446X32OCXZ3RKCYETANCNFSM4LVCESMQ > > . > — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#64 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLVRYT2BCKU3BR7STVKSL3RKCZALANCNFSM4LVCESMQ> .

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: Shrinivas.Moorthi@noaa.gov Phone: (301) 683-3718 Fax: (301) 683-3718

DeniseWorthen · 2020-03-30T15:29:40Z

So @junwang-noaa, you're suggesting we should remove the default rebuild setting from nemscommon.input and re-insert it to the individual app build files but only for the 4-component cases. I would agree w/ that approach.

climbfuji · 2020-03-30T15:30:14Z

Using compile.sh is way simpler!
…

Tell me ...

JessicaMeixner-NOAA · 2020-03-30T15:33:06Z

@junwang-noaa Yes, we have to have a clean build every time when we build the
model for the regression tests? For debugging, I've been able to turn this off, but when making/running the regression tests you have to keep this on (like in ufs-weather-model), which is an inconvenience. We could use the 4 component exe for the 3 component tests in the future to cut down on the number of builds.

climbfuji · 2020-03-30T15:35:34Z

@junwang-noaa Yes, we have to have a clean build every time when we build the
model for the regression tests? For debugging, I've been able to turn this off, but when making/running the regression tests you have to keep this on (like in ufs-weather-model), which is an inconvenience. We could use the 4 component exe for the 3 component tests in the future to cut down on the number of builds.

I suggest no further trials and experiments with this PR, except for running a final round of regression tests after updating the baseline location and finishing the code review. It will be way cheaper and quicker to experiment with this after removing the IPD PROD, IPD REPRO and CCPP REPRO tests.

SMoorthi-emc · 2020-03-30T15:38:59Z

You need updated compil.sh - see my ufs-s2s-model branch "SM_Feb252020" under tests directory. All you have to do is then "compile.sh $yourdir/FV3 hera.intel 'CCPP=Y HYBRID=N STATIC=Y SUITES=FV3_GFS_cpld_rasmgshoc MOM6=Y CICE=Y WW3=Y ' ccpp YES NO" This then can compile FV3, FV3+WW3, FV3+MOM6+CICE5, or FV3+MOM6+CICE5+WW3 and CCPP or IPD. The argument "ccpp" is not really working right now, the executable will be under "NEMS/exe" I modified the conf files so that we only need one conf file and one module file; no need to carry duplicate files. In my opinion, ufs-s2s-model and ufs-weather-model can be merged. Moorthi

…

On Mon, Mar 30, 2020 at 11:30 AM Dom Heinzeller ***@***.***> wrote: Using compile.sh is way simpler! … <#m_2702153625981825713_> Tell me ... — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#64 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLVRYT4RHQYUK5PAPOBKETRKC3JNANCNFSM4LVCESMQ> .

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: Shrinivas.Moorthi@noaa.gov Phone: (301) 683-3718 Fax: (301) 683-3718

junwang-noaa · 2020-03-30T15:50:13Z

Thanks, Moorthi. That's great if we can have one compile.sh (also same module load file) for both weather and s2s applications! So what is the "ccpp" for in your compile.sh? On Mon, Mar 30, 2020 at 11:39 AM SMoorthi-emc <notifications@github.com> wrote:

…

You need updated compil.sh - see my ufs-s2s-model branch "SM_Feb252020" under tests directory. All you have to do is then "compile.sh $yourdir/FV3 hera.intel 'CCPP=Y HYBRID=N STATIC=Y SUITES=FV3_GFS_cpld_rasmgshoc MOM6=Y CICE=Y WW3=Y ' ccpp YES NO" This then can compile FV3, FV3+WW3, FV3+MOM6+CICE5, or FV3+MOM6+CICE5+WW3 and CCPP or IPD. The argument "ccpp" is not really working right now, the executable will be under "NEMS/exe" I modified the conf files so that we only need one conf file and one module file; no need to carry duplicate files. In my opinion, ufs-s2s-model and ufs-weather-model can be merged. Moorthi On Mon, Mar 30, 2020 at 11:30 AM Dom Heinzeller ***@***.***> wrote: > Using compile.sh is way simpler! > … <#m_2702153625981825713_> > > Tell me ... > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > < #64 (comment) >, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/ALLVRYT4RHQYUK5PAPOBKETRKC3JNANCNFSM4LVCESMQ > > . > -- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: ***@***.*** Phone: (301) 683-3718 Fax: (301) 683-3718 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#64 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI7D6TPOWUDBOCX7G2X7NDLRKC4KFANCNFSM4LVCESMQ> .

DeniseWorthen · 2020-03-30T15:55:09Z

Could I request that we not move the conversation off the topic of this PR for now?

SMoorthi-emc · 2020-03-30T15:58:48Z

For now it has no use, just a placeholder. In ufs-weather model this argument copies the executable to the test directory with that name right? I didn't figure out where this copy occurs, but I guess it could be at the end of compile.sh Or, we could stay with "NEMS.x" and forget about that argument.

…

On Mon, Mar 30, 2020 at 11:50 AM Jun Wang ***@***.***> wrote: Thanks, Moorthi. That's great if we can have one compile.sh (also same module load file) for both weather and s2s applications! So what is the "ccpp" for in your compile.sh? On Mon, Mar 30, 2020 at 11:39 AM SMoorthi-emc ***@***.***> wrote: > You need updated compil.sh - see my ufs-s2s-model branch "SM_Feb252020" > under tests directory. All you have to do is then > "compile.sh $yourdir/FV3 hera.intel 'CCPP=Y HYBRID=N STATIC=Y > SUITES=FV3_GFS_cpld_rasmgshoc MOM6=Y CICE=Y WW3=Y ' ccpp YES NO" > This then can compile FV3, FV3+WW3, FV3+MOM6+CICE5, or FV3+MOM6+CICE5+WW3 > and CCPP or IPD. > The argument "ccpp" is not really working right now, the executable will be > under "NEMS/exe" > I modified the conf files so that we only need one conf file and one module > file; no need to carry duplicate files. > In my opinion, ufs-s2s-model and ufs-weather-model can be merged. > Moorthi > > On Mon, Mar 30, 2020 at 11:30 AM Dom Heinzeller < ***@***.***> > wrote: > > > Using compile.sh is way simpler! > > … <#m_2702153625981825713_> > > > > Tell me ... > > > > — > > You are receiving this because you commented. > > Reply to this email directly, view it on GitHub > > < > #64 (comment) > >, > > or unsubscribe > > < > https://github.com/notifications/unsubscribe-auth/ALLVRYT4RHQYUK5PAPOBKETRKC3JNANCNFSM4LVCESMQ > > > > . > > > > > -- > Dr. Shrinivas Moorthi > Research Meteorologist > Modeling and Data Assimilation Branch > Environmental Modeling Center / National Centers for Environmental > Prediction > 5830 University Research Court - (W/NP23), College Park MD 20740 USA > Tel: (301)683-3718 > > e-mail: ***@***.*** > Phone: (301) 683-3718 Fax: (301) 683-3718 > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > < #64 (comment) >, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/AI7D6TPOWUDBOCX7G2X7NDLRKC4KFANCNFSM4LVCESMQ > > . > — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#64 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALLVRYT3DPKJJAU45EVA4E3RKC5ULANCNFSM4LVCESMQ> .

-- Dr. Shrinivas Moorthi Research Meteorologist Modeling and Data Assimilation Branch Environmental Modeling Center / National Centers for Environmental Prediction 5830 University Research Court - (W/NP23), College Park MD 20740 USA Tel: (301)683-3718 e-mail: Shrinivas.Moorthi@noaa.gov Phone: (301) 683-3718 Fax: (301) 683-3718

junwang-noaa · 2020-03-30T16:01:15Z

Jessica, I see your point. So we do want to have clean build for RT tests. Can you clarify if the fv3-mom6-cice5-ww3 rebuild is working correctly or not?

…

On Mon, Mar 30, 2020 at 11:55 AM Denise Worthen ***@***.***> wrote: Could I request that we not move the conversation off the topic of this PR for now? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#64 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI7D6TPF5KWILV5VIFAYII3RKC6GZANCNFSM4LVCESMQ> .

JessicaMeixner-NOAA · 2020-03-30T16:04:53Z

@junwang-noaa if you use the "rebuild" option everything works fine. If you do not use the "rebuild" option we are still having issues when using WW3 and I have not tracked down that issue or resolved it yet.

junwang-noaa · 2020-03-30T16:26:59Z

I am confused, are you saying we should build with "rebuild" for fv3-mom6-cice5-ww3 exe? Currently in Dom's code: target="@[plat%EXECrt]/fv3_mom6_cice_ww3.exe" modules.nems="@[plat%INCrt]/modules_fv3_mom6_cice_ww3" md5sum="@[target].md5" # The NEMSAppBuilder is another embedded bash script in nemscommon.input build=NEMSAppBuilder(NEMS.x="@[target]",modules.nems="@[modules.nems]", OPTS="app=coupledFV3_MOM6_CICE_WW3",md5sum="@[md5sum]") how does this build (without rebuild) impact the related tests with or without ww3?

…

On Mon, Mar 30, 2020 at 12:05 PM Jessica Meixner ***@***.***> wrote: @junwang-noaa <https://github.com/junwang-noaa> if you use the "rebuild" option everything works fine. If you do not use the "rebuild" option we are still having issues when using WW3 and I have not tracked down that issue or resolved it yet. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#64 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI7D6TM3K5FTK6B732XFHA3RKC7LLANCNFSM4LVCESMQ> .

compsets/all.input

JessicaMeixner-NOAA · 2020-03-30T16:49:20Z

I am confused, are you saying we should build with "rebuild" for fv3-mom6-cice5-ww3 exe? Currently in Dom's code: target="@[plat%EXECrt]/fv3_mom6_cice_ww3.exe" modules.nems="@[plat%INCrt]/modules_fv3_mom6_cice_ww3" md5sum="@[target].md5" # The NEMSAppBuilder is another embedded bash script in nemscommon.input build=NEMSAppBuilder(NEMS.x="@[target]",modules.nems="@[modules.nems]", OPTS="app=coupledFV3_MOM6_CICE_WW3",md5sum="@[md5sum]") how does this build (without rebuild) impact the related tests with or without ww3?
…

@junwang-noaa Since we have "rebuild" as the default option in compsets/nemscommon.input we do not need to specify build or rebuild in build sections in compsets/fv3mom6cice5.input But if we do specify them there, then it should be there for both the fv3mom6cice5.exe and the fv3mom6cice5ww3.exe

junwang-noaa · 2020-03-30T18:55:28Z

I checked the code, currently in NEMSAppBuild, specify "rebuild" or specify nothing are the same, the code will build from the beginning. So adding "rebuild" or not does not make difference.

…

On Mon, Mar 30, 2020 at 12:49 PM Jessica Meixner ***@***.***> wrote: I am confused, are you saying we should build with "rebuild" for fv3-mom6-cice5-ww3 exe? Currently in Dom's code: ***@***.***%EXECrt]/fv3_mom6_cice_ww3.exe" ***@***.***%INCrt]/modules_fv3_mom6_cice_ww3" ***@***.***.md5" # The NEMSAppBuilder is another embedded bash script in nemscommon.input ***@***.******@***.***", ***@***.***") how does this build (without rebuild) impact the related tests with or without ww3? … <#m_5530479515465897229_> @junwang-noaa <https://github.com/junwang-noaa> Since we have "rebuild" as the default option in compsets/nemscommon.input we do not need to specify build or rebuild in build sections in compsets/fv3mom6cice5.input But if we do specify them there, then it should be there for both the fv3mom6cice5.exe and the fv3mom6cice5ww3.exe — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#64 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI7D6TMHLWOEJWCT55XNOTDRKDESBANCNFSM4LVCESMQ> .

DeniseWorthen

Looks good. Thanks for all the effort to create multiple RTs.

compsets/all.input

climbfuji · 2020-03-30T20:59:42Z

@DeniseWorthen thanks for checking the PR thoroughly, in particular the issue about defining cpl_defaults twice. I am just about to finish the full suite of tests after I made the changes for the "rebuild" and updated the regression test baseline to point back to the official one.

climbfuji · 2020-03-30T21:14:54Z

Regression tests on hera.intel using NEMSCompsetRun all passed (after removing the redundant "rebuild" flags and changing the regression test baseline back to the official location, i.e. for the code as it is now in the PR). Logs updated in the repository.

NEMSCompsetRun.log

This PR is ready to merge from my side.

junwang-noaa · 2020-03-31T00:04:15Z

Thank you all! Let's wait for the 8 long run cases to finish, if there is no problem there, I think we can commit the code.

…

On Mon, Mar 30, 2020 at 8:01 PM BinLi-NOAA ***@***.***> wrote: ***@***.**** approved this pull request. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#64 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI7D6TOKLQ22T3BFKTQSRXLRKEXFPANCNFSM4LVCESMQ> .

climbfuji · 2020-03-31T11:43:50Z

Thank you all! Let's wait for the 8 long run cases to finish, if there is no problem there, I think we can commit the code.
…

@JessicaMeixner-NOAA if your runs are stuck in the queue I can help, just need your instructions or copy the already set up directories.

…component and need to remove %DEBUG=Y decorator from FV3 or FV3_MAKEOPT is ignored

linlin-pan and others added 6 commits December 2, 2019 21:43

added ccpp to the coupled system

69fbaf0

Merge pull request ufs-community#1 from ufs-community/develop

9b90d1f

update

adding ccpp into s2s coupled system

088373b

compsets/all_ccpp.input compsets/cpld_fv3_ccpp_mom6_cice_2d_atm_flux.input compsets/cpld_fv3_ccpp_mom6_cice_cold_atm_flux.input compsets/fv3mom6cice5_ccpp.input coupledFV3_CCPP_MOM6_CICE.appBuilder

Merge branch 'develop' of https://github.com/panll/ufs-s2s-model into…

bdf0f6f

… develop

Update .gitmodules and submodule pointer for fv3atm for code review a…

633a33d

…nd testing

Add IPD REPRO, CCPP REPRO, CCPP PROD tests for NEMSCompsetRun

21bb7f2

climbfuji mentioned this pull request Mar 27, 2020

added ccpp to the coupled system #7

Closed

junwang-noaa requested review from binli2337 and DeniseWorthen March 27, 2020 15:03

climbfuji added 5 commits March 27, 2020 12:39

Bugfix, correct regression test baselines for several tests

07c710f

Update parm/REGTEST-FINGERPRINT.md and submodule pointer for fv3atm

e1f7ce8

For warm start tests, look for mediator files in RESTART directories …

4f60a1b

…of regression test baseline

Activate CCPP REPRO tests for verification against own baseline

794fee2

Python script create_baseline.py (temporary) for creating baseline di…

881e67c

…rectories iteratively from rtgen.PID directories

climbfuji marked this pull request as ready for review March 30, 2020 01:57

Revert change to .gitmodules following the merge of PR ufs-community#86…

9e39c84

… (Regain bit-for-bit identical results between IPD and CCPP for coupled model runs)

JessicaMeixner-NOAA reviewed Mar 30, 2020

View reviewed changes

Update regression test baseline location on hera; remove redunant 're…

d179cc3

…build' in NEMSAppBuilder commands

junwang-noaa mentioned this pull request Mar 30, 2020

negative precipitation with stochastic physics #58

Closed

DeniseWorthen reviewed Mar 30, 2020

View reviewed changes

compsets/all.input Show resolved Hide resolved

DeniseWorthen approved these changes Mar 30, 2020

View reviewed changes

compsets/all.input Show resolved Hide resolved

Regression test logs from NEMSCompsetRun on hera.intel

a639b5f

binli2337 approved these changes Mar 31, 2020

View reviewed changes

junwang-noaa mentioned this pull request Mar 31, 2020

Decide CCPP configuration for coupled benchmark 4 #30

Closed

coupledFV3_CCPP_MOM6_CICE_debug.appBuilder: bugfix, need to add CCPP …

6e607d8

…component and need to remove %DEBUG=Y decorator from FV3 or FV3_MAKEOPT is ignored

This was referenced Apr 1, 2020

IPD and CCPP test comparison and drop IPD test in S2S RT #62

Closed

Set of tests for IPD v CCPP for longer runs that mimic benchmarks #47

Closed

DeniseWorthen merged commit f53e7ce into ufs-community:develop Apr 2, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Transition CCPP to ufs-s2s-model (contains #7) #64

Transition CCPP to ufs-s2s-model (contains #7) #64

climbfuji commented Mar 27, 2020 •

edited

Loading

JessicaMeixner-NOAA Mar 30, 2020

climbfuji Mar 30, 2020

climbfuji Mar 30, 2020

junwang-noaa commented Mar 30, 2020 via email

climbfuji commented Mar 30, 2020

SMoorthi-emc commented Mar 30, 2020 via email

DeniseWorthen commented Mar 30, 2020

climbfuji commented Mar 30, 2020

JessicaMeixner-NOAA commented Mar 30, 2020

climbfuji commented Mar 30, 2020

SMoorthi-emc commented Mar 30, 2020 via email

junwang-noaa commented Mar 30, 2020 via email

DeniseWorthen commented Mar 30, 2020

SMoorthi-emc commented Mar 30, 2020 via email

junwang-noaa commented Mar 30, 2020 via email

JessicaMeixner-NOAA commented Mar 30, 2020

junwang-noaa commented Mar 30, 2020 via email

JessicaMeixner-NOAA commented Mar 30, 2020

junwang-noaa commented Mar 30, 2020 via email

DeniseWorthen left a comment

climbfuji commented Mar 30, 2020

climbfuji commented Mar 30, 2020

junwang-noaa commented Mar 31, 2020 via email

climbfuji commented Mar 31, 2020

Transition CCPP to ufs-s2s-model (contains #7) #64

Transition CCPP to ufs-s2s-model (contains #7) #64

Conversation

climbfuji commented Mar 27, 2020 • edited Loading

JessicaMeixner-NOAA Mar 30, 2020

Choose a reason for hiding this comment

climbfuji Mar 30, 2020

Choose a reason for hiding this comment

climbfuji Mar 30, 2020

Choose a reason for hiding this comment

junwang-noaa commented Mar 30, 2020 via email

climbfuji commented Mar 30, 2020

SMoorthi-emc commented Mar 30, 2020 via email

DeniseWorthen commented Mar 30, 2020

climbfuji commented Mar 30, 2020

JessicaMeixner-NOAA commented Mar 30, 2020

climbfuji commented Mar 30, 2020

SMoorthi-emc commented Mar 30, 2020 via email

junwang-noaa commented Mar 30, 2020 via email

DeniseWorthen commented Mar 30, 2020

SMoorthi-emc commented Mar 30, 2020 via email

junwang-noaa commented Mar 30, 2020 via email

JessicaMeixner-NOAA commented Mar 30, 2020

junwang-noaa commented Mar 30, 2020 via email

JessicaMeixner-NOAA commented Mar 30, 2020

junwang-noaa commented Mar 30, 2020 via email

DeniseWorthen left a comment

Choose a reason for hiding this comment

climbfuji commented Mar 30, 2020

climbfuji commented Mar 30, 2020

junwang-noaa commented Mar 31, 2020 via email

climbfuji commented Mar 31, 2020

climbfuji commented Mar 27, 2020 •

edited

Loading