Encountered segmentation fault while running the HAFS

[kbrrhari]

Dear All,
I am trying to set up the HAFS model for the Indian Ocean region. I have downloaded the source code from the hafsv1_merge branch https://github.com/hafs-community/HAFS/tree/feature/hafsv1_merge/.
In the current setup, assimilation and WW3 are turned off, and it is an ATM+OCN setup.
I am trying to simulate the Asani cyclone which is a Severe Cyclonic Storm (i.e., 07 to 11 May 2022) formed over the Bay of Bengal. While running the HAFS forecast part, I am getting a segmentation fault as shown below.
Rayleigh_Super E-folding time (mb days):
1 3.082879 5.229750
2 5.872846 6.682628
3 9.423779 9.452765
4 14.23040 15.09573
5 20.02807 26.81961
6 26.32803 52.73809
7 33.12969 125.6176
8 40.42937 466.7823
9 48.22218 16020.38
HYCOM_Run, start...w0,w1,nstep.. 1.000 0.000 31912562 31912560.0 31912563.0
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
libifcore.so.5 00002AAB72C1CA62 for__signal_handl Unknown Unknown
libpthread-2.22.s 00002AAB744A6B10 Unknown Unknown Unknown
hafs_forecast.x 00000000031CC20C fv_sat_adj_mp_fv_ 1248 fv_sat_adj.F90
hafs_forecast.x 00000000031C893D fv_sat_adj_mp_fv_ 336 fv_sat_adj.F90
hafs_forecast.x 0000000002B1E1D4 fdlmp_tedmf_nonss 166 ccpp_FV3_HAFS_v1_gfdlmp_tedmf_nonsst_fast_physics_cap.F90
hafs_forecast.x 0000000002A01351 ccpp_static_api_m 1037 ccpp_static_api.F90
hafs_forecast.x 000000000272CFFC fv_mapz_mod_mp_la 843 fv_mapz.F90
libiomp5.so 00002AAB728CA103 __kmp_invoke_micr Unknown Unknown
libiomp5.so 00002AAB7288BF3B __kmp_fork_call Unknown Unknown
libiomp5.so 00002AAB7284BE80 _kmpc_fork_call Unknown Unknown
hafs_forecast.x 000000000271CB35 fv_mapz_mod_mp_la 683 fv_mapz.F90
hafs_forecast.x 0000000002707973 fv_dynamics_mod_m 772 fv_dynamics.F90
hafs_forecast.x 0000000002657405 atmosphere_mod_mp 681 atmosphere.F90
hafs_forecast.x 00000000020AA6ED atmos_model_mod_m 838 atmos_model.F90
hafs_forecast.x 0000000001F78814 module_fcst_grid 1200 module_fcst_grid_comp.F90
hafs_forecast.x 0000000000AB158D _ZN5ESMCI6FTable1 2167 ESMCI_FTable.C
hafs_forecast.x 0000000000AB5B07 ESMCI_FTableCallE 824 ESMCI_FTable.C
hafs_forecast.x 000000000118BF18 _ZN5ESMCI3VMK5ent 2320 ESMCI_VMKernel.C
hafs_forecast.x 0000000000934F04 ZN5ESMCI2VM5ente 1216 ESMCI_VM.C
hafs_forecast.x 0000000000AB2D80 c_esmc_ftablecall 981 ESMCI_FTable.C
hafs_forecast.x 00000000005C9371 esmf_compmod_mp_e 1223 ESMF_Comp.F90
hafs_forecast.x 0000000000CAE282 esmf_gridcompmod 1896 ESMF_GridComp.F90
hafs_forecast.x 0000000001F6726F fv3gfs_cap_mod_mp 1022 fv3_cap.F90
hafs_forecast.x 0000000001F66E82 fv3gfs_cap_mod_mp 977 fv3_cap.F90
hafs_forecast.x 00000000008F208F _ZNK5ESMCI13Metho 377 ESMCI_MethodTable.C

I have given ‘ulimit -s unlimited’ as I have seen in some discussions, but the segmentation fault still persists.
I am using the default “FORECAST_RESOURCES” as mentioned in “hafsv1_hfsa.conf” file (i.e., FORECAST_RESOURCES_1080PE). To run this, I have used 60 nodes (per node 18 cores) in CRAY XC 40 machine.
Please let me know how can I resolve this error. Your help is highly appreciated.