Transition to Thompson Microphysics for Microwave All-sky Assimilation

[azadeh-gh]

Description
Transition from using the GFDL microphysics scheme, currently in operation, to the Thompson scheme (GFSv17) is proposed for the microwave all-sky assimilation. This transition aims to enhance the accuracy and reliability of forecasting. Issue#719

Resolves #719

Type of change

Please delete options that are not relevant.

• New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• New and existing tests pass with my changes
• Any dependent changes have been merged and published

[RussTreadon-NOAA]

Check for GSI Code Standard compliance. Minor items noted. Did not fully check crtm_interface.f90.

[RussTreadon-NOAA]

Thank you, @azadeh-gh , for updating the code. It looks much better now in terms of GSI code standards.

[RussTreadon-NOAA]

Rebuilt azadeh-gh:feature/ThompsonMP_Azadeh at c24e568 on Cactus and rerun ctests. Channels 1 to 13 of gmi_gpm now also shows contrl versus updat differences. AMSU-A differences occur in channels 1 to 5 and 15. ATMS differences occur in channels 1 to 6 and 16 to 22.

[xincjin-NOAA]

looks good to me.

[RussTreadon-NOAA]

Just a few minor corrections required by GSI code standards & NCO implementation standards.

[azadeh-gh]

Thank you @RussTreadon-NOAA, I removed the unused variables.

[RussTreadon-NOAA]

WCOSS ctests
Install feature/ThompsonMP_Azadeh at 1daf07b on Cactus. Run ctests with following results

Test project /lfs/h2/emc/da/noscrub/russ.treadon/git/gsi/pr743/build
    Start 1: global_4denvar
    Start 2: rtma
    Start 3: rrfs_3denvar_rdasens
    Start 4: hafs_4denvar_glbens
    Start 5: hafs_3denvar_hybens
    Start 6: global_enkf
1/6 Test #3: rrfs_3denvar_rdasens .............   Passed  729.10 sec
2/6 Test #6: global_enkf ......................***Failed  971.56 sec
3/6 Test #5: hafs_3denvar_hybens ..............   Passed  1273.47 sec
4/6 Test #4: hafs_4denvar_glbens ..............   Passed  1333.44 sec
5/6 Test #1: global_4denvar ...................***Failed  2102.97 sec
6/6 Test #2: rtma .............................   Passed  2111.63 sec

67% tests passed, 2 tests failed out of 6

Total Test time (real) = 2111.65 sec

The following tests FAILED:
          1 - global_4denvar (Failed)
          6 - global_enkf (Failed)

The global_enkf failure is expected.

The global_4denvar value is due to different analysis results. Is this expected given the change made to global_anavinfo.l127.txt? Both the updat and contrl gsi.x use the updated anavinfo file. Is the contrl gsi.x (ie, develop) lacking code to properly handle the changes to anavinfo?

[emilyhcliu]

WCOSS ctests Install feature/ThompsonMP_Azadeh at 1daf07b on Cactus. Run ctests with following results

Test project /lfs/h2/emc/da/noscrub/russ.treadon/git/gsi/pr743/build
    Start 1: global_4denvar
    Start 2: rtma
    Start 3: rrfs_3denvar_rdasens
    Start 4: hafs_4denvar_glbens
    Start 5: hafs_3denvar_hybens
    Start 6: global_enkf
1/6 Test #3: rrfs_3denvar_rdasens .............   Passed  729.10 sec
2/6 Test #6: global_enkf ......................***Failed  971.56 sec
3/6 Test #5: hafs_3denvar_hybens ..............   Passed  1273.47 sec
4/6 Test #4: hafs_4denvar_glbens ..............   Passed  1333.44 sec
5/6 Test #1: global_4denvar ...................***Failed  2102.97 sec
6/6 Test #2: rtma .............................   Passed  2111.63 sec

67% tests passed, 2 tests failed out of 6

Total Test time (real) = 2111.65 sec

The following tests FAILED:
          1 - global_4denvar (Failed)
          6 - global_enkf (Failed)

The global_enkf failure is expected.

The global_4denvar value is due to different analysis results. Is this expected given the change made to global_anavinfo.l127.txt? Both the updat and contrl gsi.x use the updated anavinfo file. Is the contrl gsi.x (ie, develop) lacking code to properly handle the changes to anavinfo?

Yes, you are correct. The global_4denvar results will be identical if both develop and the updated code run with the original global_anavinfo.l127.txt (no ni and nl in anavinfo).
Here is the regression result using the original anavinfo.

[RussTreadon-NOAA]

We need to run ctests on Hera and Hercules. Can you do this @azadeh-gh ? I will make a debug gsi.x run on WCOSS2. NCO runs our codes in debug mode as part of their pre-implementation testing.

[azadeh-gh]

We need to run ctests on Hera and Hercules. Can you do this @azadeh-gh ? I will make a debug gsi.x run on WCOSS2. NCO runs our codes in debug mode as part of their pre-implementation testing.

@RussTreadon-NOAA Yes, I run ctests on Hera and Hercules and update you

[azadeh-gh]

ctests on Hera

[RussTreadon-NOAA]

WCOSS2 debug tests
Build azadeh-gh:feature/ThompsonMP_Azadeh at 1daf07b in debug mode on Cactus. Run global_4denvar and global_enkf.

The global_4denvar case is known to fail in debug mode (see issue #775). The modifications to read_cris.f90 and read_iasi.f90 developed in issue #775 allows develop gsi.x to successfully run global_4denvar to completion.

The modified read_cris.f90 and read_iasi.f90 were added to a working copy of azadeh-gh:feature/ThompsonMP_Azadeh The debug gsi.x aborted with

GLBSOI: jiter,jiterstart,jiterlast,jiterend=    1    1    2    1
forrtl: error (73): floating divide by zero
Image              PC                Routine            Line        Source
gsi.x              0000000007F2D6AB  Unknown               Unknown  Unknown
libpthread-2.31.s  00001507B280A8C0  Unknown               Unknown  Unknown
libimf.so          00001507B2910AAF  __libm_log_l9         Unknown  Unknown
gsi.x              00000000008853DC  crtm_interface_mp        2773  crtm_interface.f90
gsi.x              000000000078BEBD  crtm_interface_mp        1881  crtm_interface.f90
gsi.x              000000000560E4B1  rad_setup_mp_setu         919  setuprad.f90

Line 1881 of crtm_interface..f90 is

        call calc_thompson_reff(rho_air,h,cloud(:,ii),cloud_names(iii),cloudefr(:,ii))

Line 2773 is

           lam_i=exp(1.0_r_kind / 3.0_r_kind * log((am_i*ni(k) *gamma(mu_i + 3.0_r_kind + 1.0_r_kind))/(qx*gamma(mu_i+1.0_r_kind))))

This PR adds these lines of code to crtm_interface.f90.

@azadeh-gh , since develop gsi.x runs to completion in debug mode but azadeh-gh:feature/ThompsonMP_Azadeh gsi.x aborts, we need to

• identify the cause for the floating diving by zero
• fix the code so that the debug gsi.x runs to completion.

GSI issue #777 has been opened to document and resolve the debug gsi.x failure.

The global_enkf ctest also aborts with

forrtl: error (65): floating invalid
Image              PC                Routine            Line        Source
enkf.x             0000000004F1793B  Unknown               Unknown  Unknown
libpthread-2.31.s  0000153993EA08C0  Unknown               Unknown  Unknown
enkf.x             000000000053EA05  letkf_mp_letkf_up         279  letkf.f90
libiomp5.so        00001539952E93F3  __kmp_invoke_micr     Unknown  Unknown
libiomp5.so        000015399526D937  __kmp_fork_call       Unknown  Unknown
libiomp5.so        0000153995231533  __kmpc_fork_call      Unknown  Unknown
enkf.x             0000000000538471  letkf_mp_letkf_up         279  letkf.f90

This is odd. Line 279 of letkf.f90 is the start of a large omp section

! Loop for each horizontal grid points on this task.                                                                                 
!$omp parallel do schedule(dynamic) default(none) private(npt,nob,nobsl, &                                                         
!$omp                  nobsl2,ngrd1,corrlength,ens_tmp,coslat, &                                                                   
!$omp                  nf,vdist,obens,indxassim,indxob,maxdfs, &                                                                   
!$omp                  nn,hxens,wts_ensmean,dfs,rdiag,dep,rloc,i, &                                                                
!$omp                  oindex,deglat,dist,corrsq,nb,nlev,nanal,sresults, &                    

It's not clear exactly what the error is.

Interestingly the debug develop gsi.x fails in exactly the same way. This indicates that this particular error is likely not due to this PR. GSI issue #776 has been opened to document the debug enkf.x failure and its resolution.

[RussTreadon-NOAA]

Since resolution of GSI issue #777 may take some time, I recommend merger of this PR into develop with the understanding that issues #777 and #776 must be resolved before GFS v17 code is passed to NCO.

[azadeh-gh]

@RussTreadon-NOAA Thanks Russ, I'm working on issue #777

[RussTreadon-NOAA]

@azadeh-gh , PR #773 has been merged into develop. Please update your branch, azadeh-gh:feature/ThompsonMP_Azadeh with develop. Then we can run ctests on the updated branch and get this PR merged into develop.

[RussTreadon-NOAA]

WCOSS ctests
Install azadeh-gh:feature/ThompsonMP_Azadeh at f32295e on Cactus. Run ctests with the following results.

Test project /lfs/h2/emc/da/noscrub/russ.treadon/git/gsi/pr743/build
    Start 1: global_4denvar
    Start 2: rtma
    Start 3: rrfs_3denvar_rdasens
    Start 4: hafs_4denvar_glbens
    Start 5: hafs_3denvar_hybens
    Start 6: global_enkf
1/6 Test #3: rrfs_3denvar_rdasens .............   Passed  728.61 sec
2/6 Test #6: global_enkf ......................***Failed  914.81 sec
3/6 Test #2: rtma .............................   Passed  1090.65 sec
4/6 Test #5: hafs_3denvar_hybens ..............   Passed  1156.18 sec
5/6 Test #4: hafs_4denvar_glbens ..............   Passed  1214.54 sec
6/6 Test #1: global_4denvar ...................***Failed  1863.49 sec

67% tests passed, 2 tests failed out of 6

Total Test time (real) = 1863.50 sec

The following tests FAILED:
          1 - global_4denvar (Failed)
          6 - global_enkf (Failed)
Errors while running CTest

The global_enkf test failure is expected. The global_4denvar failure is also expected.

[RussTreadon-NOAA]

Approve.