Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failures encountered in make_sfc_climo on Derecho #947

Closed
MichaelLueken opened this issue Oct 18, 2023 · 1 comment · Fixed by #1117
Closed

Failures encountered in make_sfc_climo on Derecho #947

MichaelLueken opened this issue Oct 18, 2023 · 1 comment · Fixed by #1117
Labels
bug Something isn't working

Comments

@MichaelLueken
Copy link
Collaborator

Expected behavior

All WE2E tests should successfully pass on Derecho.

Current behavior

Four WE2E tests, that are contained in the comprehensive.derecho test suite, are failing to run on Derecho. The four tests that are failing are:

  1. custom_ESGgrid_Central_Asia_3km
  2. custom_ESGgrid_NewZealand_3km
  3. grid_CONUS_3km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta
  4. grid_RRFS_AK_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot
    These tests are failing in the make_sfc_climo task with the following error message:
PE 8: process_vm_readv copy size mismatch: requested: 12348576 bytes copied 8836120 iter 0 total_bytes_copied 0 src_len 12348576
Assertion failed in file ../src/mpid/ch4/shm/cray_common/cray_common_memops.c at line 470: 0

Machines affected

This only appears to be an issue on Derecho.

Steps To Reproduce

  1. On Derecho, run any of the four above listed tests and the job will fail in make_sfc_climo due to copy size mismatch.
@MichaelLueken MichaelLueken added the bug Something isn't working label Oct 18, 2023
@MichaelLueken MichaelLueken mentioned this issue Jan 3, 2024
Merged
36 tasks
MichaelLueken added a commit to MichaelLueken/ufs-srweather-app that referenced this issue Aug 30, 2024
@MichaelLueken
[feature/hash_update] Ref ufs-community#947 - Uncomment tests that we…
f28558b 
…re previously failing in Derecho comprehensive WE2E test suite
@MichaelLueken MichaelLueken mentioned this issue Aug 30, 2024
Merged
17 tasks
@MichaelLueken MichaelLueken linked a pull request Aug 30, 2024 that will close this issue
[develop] Update ufs-weather-model hash and remove machine/modulefiles/WE2E test suites for decommissioned machines #1117
Merged
17 tasks
@MichaelLueken
Copy link
Collaborator Author

In hash f28558b, the four commented out tasks were uncommented. All comprehensive tests now successfully pass:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
2020_CAD_20240829065327                                            COMPLETE              79.41
2020_CAPE_20240829065330                                           COMPLETE              79.80
2019_hurricane_barry_20240829065331                                COMPLETE              74.65
2019_halloween_storm_20240829065333                                COMPLETE              75.47
2019_hurricane_lorenzo_20240829065334                              COMPLETE              75.23
2019_memorial_day_heat_wave_20240829065336                         COMPLETE              72.11
2020_denver_radiation_inversion_20240829065337                     COMPLETE              80.30
2020_easter_storm_20240829065338                                   COMPLETE              73.37
2020_jan_cold_blast_20240829065339                                 COMPLETE              81.24
community_20240829065340                                           COMPLETE              60.83
custom_ESGgrid_20240829065342                                      COMPLETE              42.27
custom_ESGgrid_Central_Asia_3km_20240829065343                     COMPLETE              71.60
custom_ESGgrid_IndianOcean_6km_20240829065345                      COMPLETE              43.83
custom_ESGgrid_NewZealand_3km_20240829065347                       COMPLETE             118.76
custom_ESGgrid_Peru_12km_20240829065349                            COMPLETE              63.86
custom_ESGgrid_SF_1p1km_20240829065350                             COMPLETE             333.39
custom_GFDLgrid__GFDLgrid_USE_NUM_CELLS_IN_FILENAMES_eq_FALSE_202  COMPLETE              27.64
custom_GFDLgrid_20240829065354                                     COMPLETE              26.03
deactivate_tasks_20240829065356                                    COMPLETE               2.63
get_from_AWS_ics_GEFS_lbcs_GEFS_fmt_grib2_2022040400_ensemble_2me  COMPLETE            1328.50
get_from_NOMADS_ics_FV3GFS_lbcs_FV3GFS_20240829065358              COMPLETE              35.91
grid_CONUS_25km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_202  COMPLETE              33.38
grid_CONUS_3km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_  COMPLETE             680.42
grid_RRFS_AK_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20240  COMPLETE             268.22
grid_RRFS_AK_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_20240829065405  COMPLETE             479.33
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP_20240829065  COMPLETE              66.24
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20  COMPLETE              71.50
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_2024082906  COMPLETE              66.78
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_202  COMPLETE              68.66
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_20240  COMPLETE              21.50
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot_20  COMPLETE              41.78
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              41.81
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_2024082906  COMPLETE              46.32
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP_20240829065  COMPLETE             138.12
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_202  COMPLETE              56.76
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_RAP_20240829065422  COMPLETE              34.36
grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2_20240  COMPLETE              27.77
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16_2024082906542  COMPLETE              74.72
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_RRFS_v1beta_202408290  COMPLETE              49.43
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2_202408  COMPLETE             467.33
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson  COMPLETE             627.11
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_20240829  COMPLETE             619.80
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR_20240829065  COMPLETE             687.09
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta_2024  COMPLETE             714.10
grid_RRFS_CONUScompact_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_  COMPLETE              67.99
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_HRRR_20240829  COMPLETE              60.68
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              62.20
grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_  COMPLETE              37.20
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024082  COMPLETE              62.01
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_2  COMPLETE              29.26
grid_RRFS_CONUScompact_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_2  COMPLETE             540.44
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_HRRR_202408290  COMPLETE             816.60
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta_20  COMPLETE             846.79
grid_RRFS_NA_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP_20240829065445  COMPLETE             159.48
grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_WoFS_v0_202408  COMPLETE              80.10
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_HRRR_2024082906545  COMPLETE              60.43
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0_20240829065  COMPLETE              44.75
grid_SUBCONUS_Ind_3km_ics_NAM_lbcs_NAM_suite_GFS_v16_202408290654  COMPLETE              63.58
grid_SUBCONUS_Ind_3km_ics_RAP_lbcs_RAP_suite_RRFS_v1beta_plot_202  COMPLETE              27.98
MET_ensemble_verification_only_vx_20240829065457                   COMPLETE               3.22
MET_ensemble_verification_winter_wx_20240829065501                 COMPLETE             327.98
MET_verification_only_vx_20240829065504                            COMPLETE               0.93
pregen_grid_orog_sfc_climo_20240829065507                          COMPLETE              27.63
specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS_20240829065511               COMPLETE              24.99
specify_template_filenames_20240829065513                          COMPLETE              31.10
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE           11604.70

This issue can be closed once PR #1117 has been approved and merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
ufs-community User Support
Archived in project
Milestone
No milestone
1 participant
@MichaelLueken