Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(+) Refactor of MOM_file_parser #74

Merged
merged 1 commit into from
Feb 28, 2022

Conversation

marshallward
Copy link
Member

This patch includes several minor changes to the MOM_file_parser and
supporting modules in order to accommodate stronger unit testing.

It includes the following API changes:

  • Removal of static_value from get_param

  • Redefined link_parameter and parameter_block as private

  • New functions: all_across_PEs(), any_across_PEs()

static_value was not used in any known experiments (outside of
internal GFDL testing), and the two derived types describe internal
operations within MOM_file_parser, so we do not expect any disruptions
from these changes.

A detailed summary of the changes are listed below.

  • assert() is now used to detect same files with different IO units.

    Detection of reopenend files of the same name but different IO unit
    has been changed from MOM_error(FATAL, ...) to assert(), to
    reflect that this should be a logically impossible result.

  • Bugfix: Reopened files are now reported to all PEs.

    If an open file is re-opened, then only the root PE will detect this
    and will return immediately. However, the others will proceed into
    populate_param_data and will get stuck in a broadcast waiting for
    root.

    We fix this by communicating the reopened state to all PEs and allow
    all ranks to return before re-processing the data.

    Note that this could also be resolved by allowing all ranks to track
    IO unit numbers, but for now we do not attempt to change this
    behavior.

  • newunit= used to generate parameter file IO unit

    The parameter IO unit is now generated by newunit= rather than an
    explicit search for an unused IO unit. Note that this is a Fortran
    2008 feature.

    Testing around available IO units has also been removed.

  • Removal of generic IO error handling

    Generic "IO error" tests, and corresponding err= arguments, have
    been removed in most cases. We now rely on the Fortran runtime to
    provide diagnostics on these errors, which should typically exceed any
    information that MOM6 could provide.

  • Removal of purported namelist support

    There were several blocks of code provided to support namelist syntax,
    but did not appear to be working, nor was there any known instance of
    it being used by anyone, so it has been removed.

  • #define/undef/= syntax testing across ranks

    Previously, only the root PE would test for consistency of the
    #define-like syntax, even though all ranks have this information.
    This required a second, awkwardly placed syntax test later in the
    subroutine.

    This test is redefined to run over all ranks, and the subsequent test
    has been removed.

  • define/override test reordering

    The found_override test when coupled to a #define-like declaration
    was unreachable due to the presence of an even stronger test related
    to valid syntax.

    This test has been moved to provide more detailed information about
    the nature of the error.

  • link_parameter, parameter_block defined as private

    Internal derived types of MOM_file_parser are redefined as private.
    This preserves the integrity of instances of these types, and also
    prevents creation of implicit object code required to access them
    externally.

  • Removal of static_value from get_param interface

    The static_value argument of get_param has been removed, since it
    is functionally equivalent to default. While this is an API change,
    there is no known case of anyone using this argument.

  • The param_type%doc fields are now properly deallocated after closed.

  • Quotes have been added around some filename error warnings, to help
    detect issues related to whitespace.

  • any_across_PEs and all_across_PEs

    New functions for calling any() and all() across PE ranks have
    been added. Behavior is in line with other functions, such as
    min_across_PEs.

@codecov
Copy link

codecov bot commented Feb 23, 2022

Codecov Report

Merging #74 (6f70955) into dev/gfdl (a468bee) will increase coverage by 0.00%.
The diff coverage is 41.17%.

❗ Current head 6f70955 differs from pull request most recent head 9caa701. Consider uploading reports for the commit 9caa701 to get more accurate results

Impacted file tree graph

@@            Coverage Diff            @@
##           dev/gfdl      #74   +/-   ##
=========================================
  Coverage     29.03%   29.04%           
=========================================
  Files           244      244           
  Lines         71865    71849   -16     
=========================================
  Hits          20869    20869           
+ Misses        50996    50980   -16     
Impacted Files Coverage Δ
src/core/MOM_verticalGrid.F90 70.21% <ø> (ø)
src/framework/MOM_coms.F90 55.83% <ø> (ø)
config_src/infra/FMS1/MOM_coms_infra.F90 41.73% <36.36%> (+1.17%) ⬆️
src/framework/MOM_file_parser.F90 62.82% <42.10%> (+1.74%) ⬆️
src/framework/MOM_domains.F90 50.84% <50.00%> (-0.88%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a468bee...9caa701. Read the comment docs.

field_flag = 0
if (field) field_flag = 1
call min_across_PEs(field_flag, pelist)
all_across_PEs = (field_flag < 1)
Copy link
Member

@Hallberg-NOAA Hallberg-NOAA Feb 27, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As written, the logic on lines 468 and 469 seems to be implementing the parallel version of .not.any(), and not the parallel version of all(). To correctly implement all_across_PEs() as the analog of all(), I think that line should be all_across_PEs = (field_flag > 0).

Exactly the same issue pertains to config_src/infra/FMS2/MOM_coms_infral.F90.

The fact that there are two identical routines in the two infra/FMS[12] directories, neither of which calls an any FMS infrastructure interfaces directly, suggests that these new routines should perhaps be implemented in src/framework/MOM_coms.F90.

Copy link
Member Author

@marshallward marshallward Feb 27, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like you are right. As you might have guessed, all_across_PEs was not used anywhere. Perhaps adding it was a mistake.

I believe this is better placed in the infra layer, because the conversions to integers and use of {min,max}_across_PEs reflects the absence of logical MPI collectives in FMS and is thus a uniquely FMS operation. A different framework would not have this restriction.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all_across_PEs test has been updated

Copy link
Member

@Hallberg-NOAA Hallberg-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes to MOM_file_parser.F90 seem well conceived.

However, as noted in detail in a separate comment attached to one of the relevant lines, the logic in the new all_across_PEs() routine looks to me like it is incorrect. Although any_across_PEs() is being used, I don't see where all_across_PEs() is being used, which could be how faulty logic could have slipped through. Moreover, I don't understand why the new any_across_PEs() and all_across_PEs() routines should be in the multiple config_src/infra/*/MOM_coms_infra.F90 files, rather than being implemented just once directly in src/framework/MOM_coms.F90.

Copy link
Member

@Hallberg-NOAA Hallberg-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR is not compiling in static memory mode. The static_value= argument at https://github.com/NOAA-GFDL/MOM6/blob/dev/gfdl/src/core/MOM_verticalGrid.F90#:~:text=call%20get_param(param_file%2C%20mdl%2C%20%22NK,static_value%3DNK_) (i.e. at about line 145 of core/MOM_verticalGrid.F90) needs to be changed to `default='.

This patch includes several minor changes to the MOM_file_parser and
supporting modules in order to accommodate stronger unit testing.

It includes the following API changes:

- Removal of `static_value` from `get_param`

- Redefined `link_parameter` and `parameter_block` as private

- New functions: `all_across_PEs()`, `any_across_PEs()`

`static_value` was not used in any known experiments (outside of
internal GFDL testing), and the two derived types describe internal
operations within `MOM_file_parser`, so we do not expect any disruptions
from these changes.

A detailed summary of the changes are listed below.

- `assert()` is now used to detect same files with different IO units.

  Detection of reopenend files of the same name but different IO unit
  has been changed from `MOM_error(FATAL, ...)` to `assert()`, to
  reflect that this should be a logically impossible result.

- Bugfix: Reopened files are now reported to all PEs.

  If an open file is re-opened, then only the root PE will detect this
  and will `return` immediately.  However, the others will proceed into
  `populate_param_data` and will get stuck in a broadcast waiting for
  root.

  We fix this by communicating the reopened state to all PEs and allow
  all ranks to return before re-processing the data.

  Note that this could also be resolved by allowing all ranks to track
  IO unit numbers, but for now we do not attempt to change this
  behavior.

- `newunit=` used to generate parameter file IO unit

  The parameter IO unit is now generated by `newunit=` rather than an
  explicit search for an unused IO unit.  Note that this is a Fortran
  2008 feature.

  Testing around available IO units has also been removed.

- Removal of generic IO error handling

  Generic "IO error" tests, and corresponding `err=` arguments, have
  been removed in most cases.  We now rely on the Fortran runtime to
  provide diagnostics on these errors, which should typically exceed any
  information that MOM6 could provide.

- Removal of purported `namelist` support

  There were several blocks of code provided to support namelist syntax,
  but did not appear to be working, nor was there any known instance of
  it being used by anyone, so it has been removed.

- `#define/undef/=` syntax testing across ranks

  Previously, only the root PE would test for consistency of the
  #define-like syntax, even though all ranks have this information.
  This required a second, awkwardly placed syntax test later in the
  subroutine.

  This test is redefined to run over all ranks, and the subsequent test
  has been removed.

- `define/override` test reordering

  The `found_override` test when coupled to a `#define`-like declaration
  was unreachable due to the presence of an even stronger test related
  to valid syntax.

  This test has been moved to provide more detailed information about
  the nature of the error.

- `link_parameter`, `parameter_block` defined as private

  Internal derived types of `MOM_file_parser` are redefined as private.
  This preserves the integrity of instances of these types, and also
  prevents creation of implicit object code required to access them
  externally.

- Removal of `static_value` from `get_param` interface

  The `static_value` argument of `get_param` has been removed, since it
  is functionally equivalent to `default`.  While this is an API change,
  there is no known case of anyone using this argument.

- The `param_type%doc` fields are now properly deallocated after closed.

- Quotes have been added around some filename error warnings, to help
  detect issues related to whitespace.

- `any_across_PEs` and `all_across_PEs`

   New functions for calling `any()` and `all()` across PE ranks have
   been added.  Behavior is in line with other functions, such as
   `min_across_PEs`.
@marshallward marshallward force-pushed the file_parser_refactor branch 2 times, most recently from ae156c2 to 9caa701 Compare February 28, 2022 14:16
@marshallward
Copy link
Member Author

Thank you for detecting this error, I've updated the PR.

@Hallberg-NOAA
Copy link
Member

This PR has passed pipeline testing at https://gitlab.gfdl.noaa.gov/ogrp/MOM6/-/pipelines/14861 .

@Hallberg-NOAA Hallberg-NOAA merged commit cf193a8 into NOAA-GFDL:dev/gfdl Feb 28, 2022
@marshallward marshallward deleted the file_parser_refactor branch March 8, 2022 19:48
adcroft pushed a commit to adcroft/MOM6 that referenced this pull request Mar 9, 2022
* additions for stochastic physics and ePBL perts

* cleanup of code and enhancement of ePBL perts

* Update MOM_diabatic_driver.F90

remove conflict with dev/emc

* Update MOM_diabatic_driver.F90

further resolve conflict

* Update MOM_diabatic_driver.F90

put id_sppt_wts, etc back.

* add stochy_restart writing to mom_cap

* additions for stochy restarts

* clean up debug statements

* clean up code

* fix non stochastic ePBL calculation

* re-write of stochastic code to remove CPP directives

* remove blank link in MOM_diagnostics

* clean up MOM_domains

* make stochastics optional

* correct coupled_driver/ocean_model_MOM.F90 and other cleanup

* clean up of code for MOM6 coding standards

* remove stochastics container

* revert MOM_domains.F90

* clean up of mom_ocean_model_nuopc.F90

* remove PE_here from mom_ocean_model_nuopc.F90

* remove debug statements

* stochastic physics re-write

* move stochastics to external directory

* doxygen cleanup

* add write_stoch_restart_ocn to MOM_stochastics

* add logic to remove incrments from restart if outside IAU window

* revert logic wrt increments

* add comments

* update to gfdl 20210806 (NOAA-GFDL#74)

* remove white space and fix comment

* Update MOM_oda_incupd.F90

remove unused index bounds, and fix sum_h2 loop.

Co-authored-by: pjpegion <Philip.Pegion@noaa.gov>
Co-authored-by: Marshall Ward <marshall.ward@noaa.gov>

* Fussing with zotero.bib.

Getting a warning about a repeated bibliography entry for adcroft2004.
Rob thinks this is a hash failure.

* Still fussing with zotero.bib

- it was complaining about the (unused) Kasahara reference.

* Several little things, one is making sponge less verbose.

- Pointing to OBC wiki file from the lateral parameterizations doc.

- Using the MOM6 verbosity to control the time_interp verbosity.

- Making the check for negative water depths more informative.

* return a more accurate error message in MOM_stochasics

* Working on boundary layer docs.

* Done with EPBL docs?

* Undoing some patches from others

* Cleaning up too-new commits

* Adding in that SAL commit again.

* correction on type in directory name

* Added some to vertical viscisity doc.

* Cleaned up whitespace leftover from porous topomerge.

- Spacing within expressions was uneven and made multiplation look like
  POW functions. Leftover from merging NOAA-GFDL#3.
- No answer changes.

* Fix out-of-bounds k index in PPM flux

- An errant use of the porous face area led to an out-of-bounds k-index
  reported in NOAA-GFDL#19.
- Closes NOAA-GFDL#19

* Adding Channel drag figure

* Take cite out of figure caption.

* Copyright year 2022
Hallberg-NOAA pushed a commit that referenced this pull request Apr 22, 2022
* remove white space and fix comment

* Update MOM_oda_incupd.F90

remove unused index bounds, and fix sum_h2 loop.

Co-authored-by: pjpegion <Philip.Pegion@noaa.gov>
Co-authored-by: Marshall Ward <marshall.ward@noaa.gov>
Hallberg-NOAA pushed a commit that referenced this pull request Apr 22, 2022
* remove white space and fix comment

* Update MOM_oda_incupd.F90

remove unused index bounds, and fix sum_h2 loop.

Co-authored-by: pjpegion <Philip.Pegion@noaa.gov>
Co-authored-by: Marshall Ward <marshall.ward@noaa.gov>
marshallward added a commit that referenced this pull request May 3, 2022
* remove white space and fix comment

* Update MOM_oda_incupd.F90

remove unused index bounds, and fix sum_h2 loop.

Co-authored-by: pjpegion <Philip.Pegion@noaa.gov>
Co-authored-by: Marshall Ward <marshall.ward@noaa.gov>
marshallward added a commit that referenced this pull request May 3, 2022
Note that most of these commits are from previously squashed pull
requests, and this PR is restoring them.

- 6360dbb Merge branch 'main' into main_to_dev
- bac8031 Merge pull request mom-ocean#1566 from jiandewang/EMC-FMS-mixed-mode-20220411
- e532d86 Merge pull request #88 from marshallward/missing_attrib_with_class_bugfix
- d380f1d An alternate fix to class(*) issues with FMS 2022-01
- 8ecf333 Merge pull request #87 from jiandewang/feature/update-to-main-20220317
- ba37f94 Merge remote-tracking branch 'FSU/main' into feature/update-to-main-20220317 this is corresponding to MOM6 main 20220317 commit (hash # 399a7db)
- 44313d9 Merge pull request #85 from jiandewang/feature/update-to-main-20220217
- 966707f Merge remote-tracking branch 'GFDL/main' into feature/update-to-main-20220217 this is corresponding to MOM6 main branch 20220217 commit (hash # 6f6d4d6), which originally based on GFDL-candidate-20220129
- 32c0e1e Merge pull request #81 from jiandewang/feature/update-to-main-20211220
- 9642b1d delete external/OCEAN_stochastic_phyiscs directory as Phil re-coded in external/stochastic_physics directory
- e7c9ada solve minor conflict in mom_cap.F90 mom_ocean_model_nuopc.F90 and MOM_energetic_PBL.F90, add two new files: src/parameterizations/stochastic/MOM_stochastics.F90 and config_src/external/stochastic_physics/stochastic_physics.F90
- 90d5961 Merge pull request #78 from jiandewang/feature/update-to-GFDL-20211019
- fd02017 Merge remote-tracking branch 'GFDL/main' into feature/update-to-GFDL-20211019
- 36f17eb Merge pull request #72 from pjpegion/ocn_stoch_july2021
- a9a957e return a more accurate error message in MOM_stochasics
- 56bb41e Merge branch 'ocn_stoch_july2021' of https://github.com/pjpegion/MOM6 into ocn_stoch_july2021
- ca2ae1c update to dev/emc
- 14ca4a1 Merge pull request #76 from jiandewang/feature/update-to-GFDL-20210914
- 29016c2 Merge remote-tracking branch 'GFDL/main' into feature/update-to-GFDL-20210914 merge GFDL main 20210914 commit (hash # c09e199)
- a8577df Merge branch 'NOAA-EMC:dev/emc' into ocn_stoch_july2021
- f8a8e4c update to gfdl 20210806 (#74)
- 16e6af0 update to dev/emc
- 237a510 add comments
- 1b4273d revert logic wrt increments
- 5b2040e add logic to remove incrments from restart if outside IAU window
- c5f2b72 add write_stoch_restart_ocn to MOM_stochastics
- bdf2dc7 doxygen cleanup
- 8bc4acc move stochastics to external directory
- a3fa3a1 Merge remote-tracking branch 'upstream/dev/emc' into ocn_stoch_july2021
- e4bc007 stochastic physics re-write
- 202cbd4 update to dev/emc
- 61717ee Merge remote-tracking branch 'origin/dev/emc' into ocn_stoch
- 565e0bb remove debug statements
- a4c0411 Merge remote-tracking branch 'upstream/dev/emc' into ocn_stoch
- 689a73f remove PE_here from mom_ocean_model_nuopc.F90
- 8afe969 clean up of mom_ocean_model_nuopc.F90
- 25ed4fc revert MOM_domains.F90
- b8d9888 place stochastic array in fluxes container and make SPPT specific arrays allocatable
- d984a7e remove stochastics container
- eb88219 clean up of code for MOM6 coding standards
- 6e3ea1b correct coupled_driver/ocean_model_MOM.F90 and other cleanup
- 0b99c1f make stochastics optional
- 85023f8 Merge remote-tracking branch 'upstream/dev/emc' into ocn_stoch
- 80f9f44 clean up MOM_domains
- 5443f8e remove blank link in MOM_diagnostics
- 1727d9a re-write of stochastic code to remove CPP directives
- 600ebf9 Merge remote-tracking branch 'upstream/dev/emc' into ocn_stoch
- 6bb9d0b fix non stochastic ePBL calculation
- 1d7ffa3 clean up code
- 040e1f1 Merge pull request #13 from NOAA-EMC/dev/emc
- 2cba995 Merge branch 'dev/emc' into ocn_stoch
- 1dc0f4f Merge remote-tracking branch 'upstream/dev/emc' into dev/emc
- 4bd9b9e clean up debug statements
- 25ed5ef additions for stochy restarts
- a2a374b add stochy_restart writing to mom_cap
- 0c15f4c Update MOM_diabatic_driver.F90
- 167a62e Merge pull request #12 from pjpegion/dev/emc
- bd477a9 Update MOM_diabatic_driver.F90
- 7212400 Update MOM_diabatic_driver.F90
- 7de295c cleanup of code and enhancement of ePBL perts
- cd06356 Merge pull request #11 from NOAA-EMC/dev/emc
- 9896d61 Merge pull request #9 from pjpegion/dev/emc_merge
- 0a62737 Merge branch 'ocn_stoch' into dev/emc_merge
- 3cad1ba Merge pull request #8 from NOAA-EMC/dev/emc
- c2aa2a8 updates from dev/emc
- 182ef34 additions for stochastic physics and ePBL perts
- 671c714 Merge pull request #1 from NOAA-EMC/dev/emc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants