Moving Nest Functionality #179
Conversation
…o that the model can run with other vertical level configurations.
…static latlon grid file once. Reduced number of variables output by mn_prog_dump_to_netcdf.
…to read from static grid files for moving nest only once. Added crude scaling of nest motion speed based on timestep.
Merge branch 'feature/hafs_moving_nest' of https://github.com/hafs-community/GFDL_atmos_cubed_sphere into feature/hafs_moving_nest Conflicts: driver/fvGFS/atmosphere.F90
…ed nest motion. Removed forced exit at timestep 4800.
…from edge, and block motion for the rest of model run.
…routines to moving nest interfaces. Added performance tracking to init_grid and setup_aligned_nest.
…hafs_moving_nest (Bill, Kyle, Xuejin, Bin)
…/fvGFS/atmosphere.F90.
*Update the moving nesting interface. Note: Coding session among Bill, Xuejin, Kyle, Zhan, Chunxi, Weiguo, Bin
…s. Added new subroutines for high precision physics variables.
*Add a fv_diag_tracker subroutine to obtain the tracker needed variables *Identify and add the call interfaces for the internal vortex tracker in driver/fvGFS/atmosphere.F90. Note: These are outcome from the coding sessions among Xuejin, Chunxi, Zhan, and Bin.
Merge branch 'feature/hafs_nest_cpl' of https://github.com/hafs-community/GFDL_atmos_cubed_sphere into feature/hafs_nest_cpl
model/fv_arrays.F90
Outdated
| @@ -1251,6 +1254,10 @@ module fv_arrays_mod | |||
| real, _ALLOCATABLE :: peln(:,:,:) _NULL !< ln(pe) | |||
| real, _ALLOCATABLE :: pkz (:,:,:) _NULL !< finite-volume mean pk | |||
|
|
|||
|
|
|||
| ! For downscaling coupling variables from parent to nested grids | |||
| real, _ALLOCATABLE :: downcpl2d(:,:) _NULL !< temporary 2d arrary for downscaling coupling variables | |||
There was a problem hiding this comment.
Does this array need to be defined in Atm% instead of being part of (say) some coupler datatype?
There was a problem hiding this comment.
I am less familiar with the coupling code. @BinLiu-NOAA can you comment here? I think this is also applicable to static nesting, with ocean coupling at the parent grid level.
There was a problem hiding this comment.
@lharris4 This downcpl2d variable is currently used to downscale/pass the SST or sea surface roughness length (z0) variables from the FV3 parent grid to its nest grid. This is because currently the ocean/wave coupling only happens between the FV3atm parent domain and the ocean/wave domains. Thus, we need to pass the SST/z0 fields from the parent to the nest so that the nest domain also gets updated values from the ocean/wave model components. This is currently a temporary solution/workaround. The plan is to enable the coupling/exchange between the FV3atm nested domain and ocean/wave domains directly in future as a better solution.
Alternatively, if you do not want add this downcpl2d variable here. We can move it back inside the subroutine atmosphere_fill_nest_cpl as a temporary array.
Please feel free to let us know your preference. Thanks!
There was a problem hiding this comment.
Hi, Bill, Bin. If it doesn't need to be preserved between calls, then it doesn't need to be in fv_array, and since it is physics-specific it probably doesn't belong here anyway.
Thanks,
Lucas
There was a problem hiding this comment.
@lharris4, Thanks for the comments/suggestions! As I was trying to move the downcpl2d array into the atmosphere_fill_nest_cpl subroutine, it seems to me that will make the procedure passing a variable from parent to its nest much more complicated. Thus, at this moment, I just renamed this downcpl2d array into parent2nest_2d array to reduce any potential confusion. And I also think the name of parent2nest_2d is actually more accurate, which is to assist the parent to nest downscaling procedure. It can be used for passing not only coupling variables but also any other desired dynamics/physics variables from the parent to its static/moving nest. Hopefully, this is fine for this temporary solution. And we can revisit this once the direct coupling between the FV3 nested grids and the wave/ocean domains are implemented/enabled. Thanks!
There was a problem hiding this comment.
Thanks. We will revisit in a future release.
parent2nest_2d variable is to downscale/remap a 2d variable from the parent to its nest.
|
Hi, Rusty. I would appreciate it if debugging code were removed.
Thanks,
Lucas
…On Fri, Apr 1, 2022 at 11:11 AM Rusty Benson ***@***.***> wrote:
***@***.**** commented on this pull request.
There is an FMS namelist option to have the diagnostic data be dumped on
write as opposed to buffered until exit.
------------------------------
In driver/fvGFS/atmosphere.F90
<#179 (comment)>
:
> + ! WDR start code to output moving nest debug information after timestep.
+ ! This is solely debugging information; not required for moving nest functionality.
+#ifdef MOVING_NEST
+
+ !call date_and_time(TIME=str_time)
+ !write(message,'("TIMESTEP atmosphere.F90 a_step=",I0," fcst_hr=",F8.2," time=",A12)') a_step, a_step * dt_atmos / 3600.0, str_time
+ !call mpp_error(NOTE,message)
+
+ !! Re-enable to output buffered NC files early in a run.
+ !if (tsvar_out .and. a_step .eq. 4800) then
+ ! call fms_io_exit() !! Force the output of the buffered NC files
+ ! print '("[INFO] WDR calling mpp_exit after moving nest atmosphere.F90 npe=",I0)', this_pe
+ ! call fv_io_exit()
+ ! call mpp_exit()
+ ! print '("[INFO] WDR calling STOP after moving nest atmosphere.F90 npe=",I0)', this_pe
+ ! stop
+ !endif
+
+ !! WDR End code
+#endif
+
@lharris4 <https://github.com/lharris4> - did you want this debugging
code removed?
------------------------------
In model/fv_control.F90
<#179 (comment)>
:
> @@ -161,6 +161,12 @@ module fv_control_mod
use molecular_diffusion_mod, only: molecular_diffusion_init, &
read_namelist_molecular_diffusion_nml
+#ifdef MOVING_NEST
+ use fms_io_mod, only: fms_io_exit
I don't see this use association being used.
------------------------------
In model/fv_grid_utils.F90
<#179 (comment)>
:
> + logical, save :: first_time = .true.
+ integer, save :: id_latlon
+
+ !if (first_time) then
+ ! id_latlon = mpp_clock_id ('latlon2xyz', flags = clock_flag_default, grain=CLOCK_ROUTINE )
+ !end if
+
+ !call mpp_clock_begin (id_latlon)
+
This can be removed.
------------------------------
In model/fv_grid_utils.F90
<#179 (comment)>
:
> + !call mpp_clock_end (id_latlon)
+
This can be removed.
------------------------------
In tools/fv_diagnostics.F90
<#179 (comment)>
:
> +#ifdef MOVING_NEST
+!use fv_moving_nest_physics_mod, only : dump_surface_physics
+!use fms_io_mod, only : fms_io_exit
+!use fv_io_mod, only : fv_io_exit
+!use mpp_mod, only : mpp_exit
+#endif
+
This can be removed.
------------------------------
In tools/fv_diagnostics.F90
<#179 (comment)>
:
> +#ifdef MOVING_NEST
+ !if ( k .eq. km) then
+! print '("[INFO] WDR ",A," RANGE error at lowest level (",I0,",",I0,",",I0,")")', qname, i, j, k
+! call dump_surface_physics(i, j, k)
+ !endif
+#endif
This can be removed.
------------------------------
In tools/fv_diagnostics.F90
<#179 (comment)>
:
> +#ifdef MOVING_NEST
+!! !! Dump FMS debugging information
+! call fms_io_exit() !! Force the output of the buffered NC files
+! print '("[INFO] WDR calling mpp_exit after moving nest atmosphere.F90 npe=",I0)', mpp_pe()
+! call fv_io_exit()
+! call mpp_exit()
+! print '("[INFO] WDR calling STOP after moving nest atmosphere.F90 npe=",I0)', mpp_pe()
+! stop
+#endif
+
This can be removed.
—
Reply to this email directly, view it on GitHub
<#179 (review)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AMUQRVBT7KN5H3MVYMQF243VC4GXTANCNFSM5QNGE35A>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Merge branch 'feature/hafs_nest_cpl' of https://github.com/hafs-community/GFDL_atmos_cubed_sphere into feature/hafs_nest_cpl
There was a problem hiding this comment.
@wramstrom Can you please create an issue logging the decisions we made in our meeting on March 23rd? I believe Lucas summarized this in a Google Doc, but it would be helpful to have that information in an issue on Github.
|
@laurenchilutti I have created the issue here: |
Great! Thank you |
|
@wramstrom - are you still planning to remove the timers from fv_grid_tools.F90 as discussed? |
@bensonr I am using the timers in fv_grid_tools.F90 in my optimization work that started this week and runs through May 1. I can remove them now if it is a significant issue, but would prefer to remove them on the next pass after I have completed performance optimization work. |
|
@JosephMouallem @bensonr @DusanJovic-NOAA Could you please leave a review? If you think it is ready to merge please approve. Thank You! |
|
@junwang-noaa this PR is now ready to merge. |
|
@wramstrom I see in the description of this PR you would like me to NOT squash commit this. Why is this? I want to just note that this is 136 commits. |
You can squash commit this if needed; @BinLiu-NOAA showed me some other ways I can examine the evolution of the code if I need it. |
|
@laurenchilutti Please merge this PR. The ufs-weather-model passed full regression tests (ufs-community/ufs-weather-model#1104). |
Moving Nest Functionality (NOAA-GFDL#179)
Description
This pull request contains the dycore changes to implement moving nest functionality, as specified in Issue #1100:
ufs-community/ufs-weather-model#1100
This code implements a storm-following moving nest in regional and global configurations. Nest motion code handles prognostic variables, surface variables, and physics variables. Nest motion is configured in the namelist, with options for automated storm tracking, prescribed nest motion, and frequency of nest moves. Automated storm tracking code was ported from HWRF to operate within the FV3 framework. This code also includes filling SST in a static or moving nest by downscaling from coupled SST in the parent domain.
Moving nest code has been designed to allow prior static global or nested simulations to have identical performance and be bitwise reproducible. This has been verified with regression testing. Moving nest code is wrapped with #ifdef MOVING_NEST in existing and new code modules. The bulk of the new code is in new files in atmos_cubed_sphere/model; the main files are fv_moving_nest_main.F90, fv_moving_nest.F90, fv_moving_nest_physics.F90, and fv_tracker.F90. Additional utilities subroutines are added in new files fv_moving_nest_utils.F90 and bounding_box.F90. The existing module fv_arrays.F90 has significant additions to enable moving physics variables and the automated storm tracking. The existing module fv_control.F90 has some smaller changes to enable moving nest, and coupling SST in a static or moving nest to an ocean model.
The current code supports a single moving nest, at 3X nest refinement. Code has been designed to be extensible to multiple moving nests.
Dependencies:
This code requires the updated FMS library from PR #840 NOAA-GFDL/FMS#840 to include mpp_shift_nest_domains().
The moving nest code can only be run when the associated PR from the FV3 level calls into the moving nest subroutines added in this PR.
Fixes #1100 ufs-community/ufs-weather-model#1100
How Has This Been Tested?
This code has been tested on Orion and Hera with Intel 18 for a large number of stress cases. These exercised the moving nest functionality in regional and global configurations. Most tests were performed with a regional configuration at C512 with a 3X refinement nest that spanned 19 degrees X 19 degrees. Tests followed storms over the open ocean, through the Caribbean islands, and during continental landfall and subsequent overland passage. Tests of land interaction are important because a large number of surface fields are specified for land-only points. In addition, interaction with sharp terrain validates the stability of the model. Additional tests were performed in the Southern Hemisphere, and crossing the dateline and prime meridian. Tests focused on the FV3_HAFS_v0_gfdlmp_tedmf physics suite.
Spreadsheet of tests performed is available at: https://docs.google.com/spreadsheets/d/1FP6JReNuCX4hQnEZI1irkooMuw_DiGvXRm1_LxJHhL4/edit#gid=0
The main regression suite now has 5 cases to test moving nest functionality; those config changes will be in the FV3 level pull request for moving nest. Tests are called: hafs_regional_specified_moving_1nest_atm, hafs_regional_storm_following_1nest_atm, hafs_regional_storm_following_1nest_atm_ocn, hafs_regional_storm_following_1nest_atm_ocn_wav, hafs_global_storm_following_1nest_atm.
Moving nest development team requests that the commits that make up this PR NOT be squashed.
The moving nest capability in the model core was coded by @wramstrom at AOML/HRD with significant collaboration from @BinLiu-NOAA at EMC, and additional collaboration from AOML/HRD (@Xuejin-Zhang, @A-Kyle, Gopal,et al.), EMC (@ChunxiZhang-NOAA, @ZhanZhang-NOAA, @WeiguoWang-NOAA, @JunghoonShin-NOAA, et al.) and GFDL (@lharris4, @bensonr, @TimMarchok-NOAA, @JosephMouallem, et al.).
Checklist:
Please check all whether they apply or not
Follows Fortran style guide, except some lines exceed 120 character length.
Main fv_moving_nest*.F90 modules have been commented. Utilities modules are not yet fully commented.