Fix wave init

[pjpegion]

Description

Fixes the logic for a restart when there are not wave in the coupler and start_type needs to be set to initial. This is needed for EP4 which will use warm starts from the replay, but the wave initial conditions are not part of the replay, so they are not in the mediator.

Top of commit queue on: TBD

Input data additions/changes

• No changes are expected to input data.
• There will be new input data.
• Input data will be updated.

Anticipated changes to regression tests:

• No changes are expected to any regression test.
• Changes are expected to the following tests:

cpld_control_p8 and cpld_restart_p8

Subcomponents involved:

• AQM
• CDEPS
• CICE
• CMEPS
• CMakeModules
• FV3
• GOCART
• HYCOM
• MOM6
• NOAHMP
• WW3
• stochastic_physics
• none

Combined with PR's (If Applicable):

Commit Queue Checklist:

• Link PR's from all sub-components involved
• Confirm reviews completed in sub-component PR's
• Add all appropriate labels to this PR.
• Run full RT suite on either Hera/Cheyenne with both Intel/GNU compilers
• Add list of any failed regression tests to "Anticipated changes to regression tests" section.

Linked PR's and Issues:

• fixes Wave model cannot restart if mediator flag is set to startup #1546
• fixes Wave model cannot restart if mediator flag is set to startup NOAA-EMC/WW3#870
• Depends on fix clock initialization for a restart NOAA-EMC/WW3#871

Testing Day Checklist:

• This PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR.
• Move new/updated input data on RDHPCS Hera and propagate input data changes to all supported systems.

Testing Log (for CM's):

• RDHPCS
  • Intel
    • Hera
    • Orion
    • Jet
    • Gaea
    • Cheyenne
  • GNU
    • Hera
    • Cheyenne
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
  • Completed
• opnReqTest
  • N/A
  • Log attached to comment

[DeniseWorthen]

@pjpegion I'm not sure I understand by "not wave in the coupler" Do you mean that the wave fields are not in the coupler restart file?

[pjpegion]

yes, it should have said waves are not in the mediator/coupler restart files.

[DeniseWorthen]

Thanks for the clarification. You said you tested w/ cpld_control and cpld_restart. But if the issue is that you don't have the wave ICs from the replay, wouldn't the test have been to run w/o waves for the control case and then try to restart with that but with waves included?

[pjpegion]

@DeniseWorthen I will do that test to confirm it works in that situation

[pjpegion]

@DeniseWorthen I did the following tests which all worked

1. run cpld_control_p8 without waves, dropping restarts at FH012 and used those restarts to initialized the 2 tests below
2. run cpld_restart_p8 with waves model and wave restart from a different run
3. run cpld_restart_p8 with wave, but no restart and wave model started at rest.

[BrianCurtis-NOAA]

I've brought in the new PR template and copied over the appropriate text. Please fix what I may have missed. This is still waiting on WW3 review of that code, and then you will need to run the full RT suite with an up-to-date (as possible) branch before this makes it into the commit queue.

[BrianCurtis-NOAA]

Also I wasn't sure if this needed new baselines, but assumed it did. Please remove that label if I am incorrect.

[DeniseWorthen]

@pjpegion I believe the fix I had proposed for adding a time-offset is cleaner than your proposed change. It leaves intact the logic of initial/continue and adds a time-offset for fhrot in the same way the driver does.

[pjpegion]

@DeniseWorthen I remember you suggesting the time offset. Did you code that up?

[DeniseWorthen]

@pjpegion yes, I did code it up. I added a few log messages regarding the clock, but those aren't really required. You can see the change in wav_comp_nuopc in this branch. For some reason I can't find your fork to compare to.

Note this branch also fixed CI for the dev/ufs-weather-model branch and updated to the latest develop so additional changes are present.

[BrianCurtis-NOAA]

Still waiting on Intel/GNU logs from Hera or Cheyenne showing the anticipated RT's are the only ones failing.

[zach1221]

Hi, @pjpegion. In anticipation of working to test this PR next, would you be able to resolve the conflicts and provide Intel/GNU logs for Hera or Cheyenne?

[JessicaMeixner-NOAA]

@pjpegion is on leave this week so I'm helping him w/resolving the conflicts and can provide the intel/gnu for hera.

[zach1221]

@JessicaMeixner-NOAA Ok, understood. Thank you!

[JessicaMeixner-NOAA]

Okay the WW3 branch for this PR has been updated. And I have a PR with the updated and merge conflicts resolved and is here: pjpegion#14

I'll post gnu/intel hera logs as soon as i have them.

[jkbk2004]

@JessicaMeixner-NOAA @pjpegion the pr still needs work. Meanwhile, we like to check next pr #1642 as well. FYI @zach1221 @BrianCurtis-NOAA

[BrianCurtis-NOAA]

@jkbk2004 #1642 isn't ready yet either.

[pjpegion]

@JessicaMeixner-NOAA cleaned up my mess.

[jkbk2004]

Automated RT Failure Notification
Machine: gaea
Compiler: intel
Job: RT
[RT] Repo location: /lustre/f2/pdata/ncep/emc.nemspara/autort/pr/1173997832/20230314014543/ufs-weather-model
[RT] Error: Test cpld_decomp_p8 006 failed in run_test failed
[RT] Error: Test cpld_debug_p8 015 failed in run_test failed
Please make changes and add the following label back: gaea-intel-RT

[JessicaMeixner-NOAA]

I re-ran the 2 failed jobs on gaea this morning and both passed: /lustre/f2/scratch/ncep/Jessica.Meixner/ufs-weather-model/tests/RegressionTests_gaea.intel.log

[zach1221]

@JessicaMeixner-NOAA thank you. I'll kill my tests running and use your logs.

[JessicaMeixner-NOAA]

@jkbk2004 let me know when you're ready for me to merge in the WW3 update.

[jkbk2004]

@JessicaMeixner-NOAA sure! go ahead to merge ww3 pr.

[JessicaMeixner-NOAA]

The WW3 PR has been merged. The hash from the sqush commit for WW3 is: af544dca5bf49bf3c3ad8d47b786e8211dcf4bba

I dont know if @pjpegion is out for the day yet or not, but if he is maybe someone w/maintainer privelages can push the revert to his branch?

[pjpegion]

I'm here, and will do the update to my branch.

[BrianCurtis-NOAA]

Hash looks good.