Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
This patch introduces an index rotation feature applied to all internal
fields. When this feature is enabled, model fields are initialized on
the index map as specified by the configuration or input files. The
values of these arrays are then moved to a rotated index map and saved
to a new rotated array, which are then used for model simulation.
The primary purpose of this feature is to detect regressions in
rotationally symmetric calculations, such as fluid dynamic dynamic
processes and other transport phenomena.
Fields created or read by the drivers are unrotated and are handled as
external user-defined field. External fields are rotated or unrotated
as they pass between the driver and the ocean model.
In order to maintain bit reproducibility, all field initialization is
computed or assigned on the input grid.
This patch also includes an extension of the test suite, which applies a
single quarter-turn to each of the existing tests.
The features of this patch are summarized below.
Two new configuration features are added:
- ROTATE_INDEX: Boolean flag which enables this feature.
- INDEX_TURNS: Integer which descibes the number of counterclockwise
quarter-turns. Default value is 1.
Two new modules and several new functions are added:
- MOM_array_transform: Fundamental operations for rotating arrays,
vectors, rotationally equivalent pairs
- MOM_transform_FMS: Wrappers to FMS operations which can only operate
on the unrotated index pax.
- Numerous rotation functions have been introduced in various modules
associated with initialization (forcing, OBC, etc), which are
documented in the source code.
The following new features have been added to existing components:
- paired checksums (`uvchksum`, `hchksum_pair`) now include a `scalar_pair`
argument to distinguish between rotationally equivalent scalars (e.g.
grid areas on u- and v-points) and vector components.
- `register_restart_pair` was introduced to register rotationally
associated pairs.
- `turns` has been added to the horizontal index (HI) type, as a means
to track the number of quarter turns in lower level operations.
- `allocate_forcing_type` and `allocate_mech_forcing` have been extended
to permit allocation based on an exisiting (unrotated) reference,
alongside the older interface using control flags for each field type.
- copy_MOM_domain includes a `turns` argument, to support rotational
changes to the FMS domain.
Reproducibility:
- All results should be bitwise reproducible. A nonphysical exception
exception is the calculation of CFL conditions in systems with zero
velocity, such as the unit test case. The previous implementation
could produce a "negative zero" CFL, whose sign would depend on the
number of rotations.
The new implementation eliminates these negative zeros, but may also
change existing CFL calculations, and may report a regression in
`ocean.stats`.
- Rotated diagnostic checksum output is not tested, since there is not
yet a way to associated rotationally equivalent diagnostics. See
below for more information.
Current limitations:
- Only single PE runs can be rotated.
Currently FMS assumes a specific ordering for assignment of PE IDs
(east->west, south->north). An index rotation will typically
re-orient these domains in a way which breaks this default
assignment. This ID is used to manage MPI message passing
targets, so any rearrangement of these IDs will cause errors.
Since we currently cannot control the PE ID assigned by FMS, we are
currently unable to support index rotation of parallel runs.
- Certain fields and tracers cannot be rotated.
There are a small subset of tracers and fields in various
structures, such as boundary fluxes, which are currently
unsupported. This is primarily beacuse these tracers are typically
managed at the driver level.
There is no reason why these fields cannot be supported, but it
would require a larger test suite to detect and handle such fields.
- Diagnostic output is not rotated.
Diagnostics are computed on the rotated grid, but it would require
additional FMS wrappers to support registering and writing
diagnostics on the input index map. This is not yet implemented,
so all diagnostics are on the rotated index map.
One additional consequence is that there is no way to register pairs
of diagnostics, and thus no way to automate the chksum.diag output.
Currently, diagnostic checksums are disabled in the test suite.
We hope to resolve this issue in a future patch.
- Open boundary condition (OBC) rotation is restriced to INDEX_TURNS=1
Generalized OBC rotation is currently not supported. This is
primarily due to the sensitivity of segment configuration, which is
specified with a large number of flags denoting its orientation.
It is possible to support generalized rotation, but it will require
a refactor of the OBC segment code.
We hope to support generalized rotation in a future patch.
- Incorrect definitions of gridLonT and gridLatT
Due to certain array length assumptions for 1d axes, we rotate and
exchange the values of gridLatT and gridLonT during quarter-turn
operations.
This is not correct, since these arrays describe latitude and
longitude, rather than the axes in the first and second dimension.
However, several other components currently expect the first and
second reference axis to have matching array lengths, so for now we
swap these arrays during rotations.
The 1d arrays are only used for initialization computation (which is
on the input map) and I/O management, so this error does not affect
any existing runs. But rotated runs which save these arrays will be
incorrect.
- Non-ALE sponge rotation is disabled.
Only ALE sponge rotation is currently supported.
It should be possible to extend rotation to include non-ALE sponges
if needed.
Further comments:
- Fields are not rotated in-place, since many calculations require that
both fields exist at any time. While it is unlikely that rotated and
unrotated fields would persist for an entire run, there is likely to
be an increase in memory overhead.
- External fields from the drivers are rotated and unrotated at the
beginning and end of every timestep, which can create additional
computational overhead.
Note that there are control flags, e.g. `cycle_start` and `cycle_end`
which could be used to manage these operations, assuming that the
driver does not change the fields inbetween coupler steps.
- This patch fixes what appears to have been an error in the global
h-point indexes of OBC segments. However, it also seems that these
indices were unused, so this is unlikely to impact existing runs.
This implementation was proposed by Robert Hallberg and follows an
earlier implementation by Nic Hannah, whose efforts introduced several
changes that implemented the rotational invariance of existing numerical
calculations.- Loading branch information
1 parent
3e9c645
commit 22215cb