-
Notifications
You must be signed in to change notification settings - Fork 119
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Stochastic physics is not working correctly after yaml/Python changes to the workflow #818
Comments
@JeffBeck-NOAA Can you point me to a case for the deterministic problem you're seeing? I have added a new deterministic SPP case for my PR #811 and I observed the expected perturbations. You can see the cases I ran on Hera: With SPP:
Without SPP:
|
@mkavulich, here are two deterministic cases on Hera, one with SPPT, one with SPPT+SPP: /scratch2/BMC/fv3lam/beck/FV3-LAM/expt_dirs/deter_SPPT Neither have anything in the input.nml nam_stochy or nam_sppperts sections. The config.yaml file for the second expt_dir (SPPT+SPP) is here (I just added DO_SPP=true): /scratch2/BMC/fv3lam/beck/FV3-LAM/ufs-srweather-app/ush/config.yaml Thanks for taking a look! |
For the ensemble date/ensmem seed problem, you can look here on Hera: /scratch2/BMC/fv3lam/beck/FV3-LAM/expt_dirs/runtime_SPPT_ens You'll see that mem001 is pointing to the input.nml_stoch file in /scratch2/BMC/fv3lam/beck/FV3-LAM/expt_dirs/runtime_SPPT_ens and not the input.nml file in the the member directory as it needs to. It's also not using the date/ensmem string in input.nml_base, which is in the mem001 directory. |
@JeffBeck-NOAA Is there a WE2E config that I can work on to see if I can find the source of the issue? |
@christinaholtNOAA, the following WE2E should illustrate the problems with stochastic physics for a deterministic configuration: grids_extrn_mdls_suites_community/config.grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_HRRR.yaml And the following WE2E should illustrate the problem with the seeds in ensemble mode and the soft-linked input.nml (should use one unique input.nml file per member): wflow_features/config.community_ensemble_2mems_stoch.yaml |
Thanks @JeffBeck-NOAA. I will try to work in a debugging session with this soon, but I might not get there right away. Thanks for taking the time above to clearly describe the problem above. It will be super helpful for diving in. |
Thanks, @christinaholtNOAA! |
I was able to identify the hash where the stochastic physics stopped working correctly - 7baa285, which is associated with PR #744. I was wondering, do we want stochastic physics to work with deterministic runs? Given the pre-existing logic in I just wanted to bring this to everyone's attention so that we can have more eyes on this. |
@MichaelLueken, thanks for digging into the hashes to identify where things changed. Stochastic physics was functional for both deterministic and ensemble runs in the shell-based SRW App when it was originally committed here and here. I'm not sure what pre-existing logic is in |
@JeffBeck-NOAA In
Before this, I'm seeing the following logic:
From the sounds of it, the For the ensembles, I think the issue is that only the
Not not clear to me where the rest of the contents for these two namelist options have gone. Once this has been addressed, I think that the ensemble issue will be cleared up. |
@MichaelLueken, you hit on the main problem. After taking a look, it was indeed the "DO_ENSEMBLE" part of scripts/exregional_run_fcst.sh and generate_FV3LAM_wflow.py that was the issue with stochastic physics not being used in the deterministic runs. I fixed that issue with changes to the ex-script and generate_FV3LAM_wflow.py. @willmayfield found the other problem with ensemble mode, where the wrong namelist was being referenced and was linked instead of copied into the member directories. We have put both of our changes into a branch on Will's fork of the App, and he will open a PR shortly. |
Expected behavior (previous correct behavior prior to the yaml/Python updates)
Current behavior
Machines affected
All machines.
Steps To Reproduce
Detailed Description of Fix (optional)
Changes to the Pythonized set_FV3nml_ens_stoch_seeds.py are likely required to fix the seed problem. Modifications to when stochastic namelist entries are applied is necessary to fix the deterministic stochastic physics problem.
Correct behavior:
Example namelist block for nam_stochy when running in deterministic mode with stochastic physics turned on (SPPT in this case):
&nam_stochy
iseed_sppt = 1
new_lscale = .true.
sppt = 0.7
sppt_logit = .true.
sppt_lscale = 150000
sppt_sfclimit = .true.
sppt_tau = 21600
spptint = 3600
use_zmtnblck = .false.
/
Example mem001 namelist block for nam_stochy when running in ensemble mode with stochastic physics turned on (SPPT in this case):
&nam_stochy
iseed_sppt = 2019061500011
new_lscale = .true.
sppt = 0.7
sppt_logit = .true.
sppt_lscale = 150000
sppt_sfclimit = .true.
sppt_tau = 21600
spptint = 3600
use_zmtnblck = .false.
/
@christinaholtNOAA @MichaelLueken @mark-a-potts @mkavulich @gsketefian @michelleharrold @willmayfield
The text was updated successfully, but these errors were encountered: