SIS2 crashes after reading huge (missing) values on land #104
Comments
|
Hi Niki,
Because reproducing_sum also will also sum over the land points. To solve
the issue, you may remove the option -m when doing combine for restart file.
Zhi
…On Tue, Apr 23, 2019 at 5:23 PM Niki Zadeh ***@***.***> wrote:
We recombined restarts of a OMIP4p5 run that was using a mask_table. The
values of all restart variables on the masked pes are set to a missing
value of 9.9692E+36 (when restarts are written by fms).
When trying to restart the model using a layout other than the one used in
producing restarts the model crashes after warnings like:
WARNING from PE 255: Bad ice state tsurf Start of set_ice_surface_state ; at 54.7 68.1 or i,j,k = 30 5 1; nbad = 23000 on pe 255 ; T_skin = 9.9692E+36, ps = 0.0000E+00
WARNING from PE 255: mi/ms = 0.0000E+00 0.0000E+00 ts = 2.4973E+33 ti = 2.4973E+33, 2.4973E+33, 2.4973E+33, 2.4973E+33
WARNING from PE 255: enth_snow = 9.9692E+36 enth_ice = 9.9692E+36, 9.9692E+36, 9.9692E+36, 9.9692E+36
WARNING from PE 255: salin_ice = 9.9692E+36, 9.9692E+36, 9.9692E+36, 9.9692E+36
FATAL from PE 19: Overflow in reproducing_sum(_3d) conversion of -5.60385+140
Note these values are the missing values on land.
Why does it matter what the land values are set to?!
workdir
/lustre/f2/scratch/Niki.Zadeh/work/xanadu_mom6_devgfdl_20190411/OM4p5_IAF_BLING_CFC_abio_csf_mle200_initCond1998_1x0m2d_256x1o.o1556049322
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#104>, or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABMQJECZHDQSJ27HPHAJDO3PR543XANCNFSM4HH6QVQQ>
.
|
|
This isn't a practical solution for existing combined restart files.
As @nikizadehgfdl points out, it seems like the ice model shouldn't care
what the values are over land. Can some sort of masking be implemented to
allow for the use of older, combined restarts?
|
|
This is doable in the source code to add one optional argument mask. But we
currently have two version of reproducing_sum, one in MOM6 and one in FMS
infrastructure. The FMS version is copied from MOM6. We may think about the
possibility that MOM6 use FMS version. In this case, we just need to change
one version.
Greetings,
Zhi
…On Wed, Apr 24, 2019 at 10:22 AM John Krasting ***@***.***> wrote:
This isn't a practical solution for existing combined restart files.
As @nikizadehgfdl points out, it seems like the ice model shouldn't care
what the values are over land. Can some sort of masking be implemented to
allow for the use of older, combined restarts?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#104 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/ABMQJEBQTPLL6HVKYUEGWCTPSBUJ5ANCNFSM4HH6QVQQ>
.
|
|
We did not use -m in mppnccombine. And I see no difference in output files using -m or not (both cases has the huge values on masked pes). For some variables like part_size I cannot reset these to arbitrary values such as 0 since the values on z_axis (6 of them) must sum to 1 and 1/6 would not do either. When I did reset huge to 1/6 inside the code answers changed. Which brings us back to why the values on land matter at all! One more clue, this happens only for mask_tables which cut too close to the land boundary. It's as if these huge values somehow get into the halos for restart variables after reading the restarts. |
|
Hi Zhi and Niki I am MOM6 +SIS2 for the NEP region. While running, I did use mask_table. I start running the model from 1993 up to 1997 model runs fine, but after following the warning showing what was going on, I didn't understand. |
|
Hi Vivek, This seems like a different sort of problem than what this issue is referring to. This point is not in the land mask of the NEP domain. Nevertheless, you don't want a temperature or salinity in the hundreds! This is at the surface - do you see anything in the surface forcing to account for it? Can you see it starting to go bad before it blows up? This is probably not a SIS2 problem per se, rather something that could be brought up in the MOM6 forum. -Kate |
|
Hi Kate, Thank you very much for your response. When I am run with climatological runoff forcing, there is no warning. This warning comes when I am using intern-annual runoff forcing. But from 1993 to 1997 runs fine, since 1998 the month of March shows this warning. Still model running 1999th year. |
|
It is still running? Does the ocean think your SSS is over 200? This point is nowhere near where the runoff should be happening. Have you looked at your surface ocean fields? |
|
Hi Kate, Thank you very much for your response. I will look into the surface ocean fields, and I will let you know. Thanks, |
We recombined restarts of a OMIP4p5 run that was using a mask_table. The values of all restart variables on the masked pes are set to a missing value of 9.9692E+36 (when restarts are written by fms).
When trying to restart the model using a layout other than the one used in producing restarts the model crashes after warnings like:
Note these values are the missing values on land.
Why does it matter what the land values are set to?!
workdir
The text was updated successfully, but these errors were encountered: