Allow APP to differ between RUNs #2943

DavidHuber-NOAA · 2024-09-19T18:25:52Z

Description

This enables APP to be specified for each RUN. This also removes the need for a _no_run configuration dictionary and somewhat simplifies the _init_finalize method.

Resolves #2908
Resolves #2956

Type of change

New feature (adds functionality)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any submodules? NO

How has this been tested?

Generated .xml files on Hera for all available CI tests and verified they were identical to develop.
Created an arbitrary APP configuration ({"gdas": "S2S", "gfs": "ATM"})

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
New and existing tests pass with my changes

DavidHuber-NOAA · 2024-09-19T19:21:42Z

This is going to require some more leg work as some APP-specific flags are currently read in without specifying a RUN:

global-workflow/workflow/applications/applications.py

Lines 34 to 50 in c6e3262

    
           base = conf.parse_config('config.base') 
        
           self.mode = base['MODE'] 
        
           if self.mode not in self.VALID_MODES: 
        
               raise NotImplementedError(f'{self.mode} is not a valid application mode.\n' 
        
                                         f'Valid application modes are:\n' 
        
                                         f'{", ".join(self.VALID_MODES)}\n') 
        
           self.net = base['NET'] 
        
           self.model_app = base.get('APP', 'ATM') 
        
           self.do_atm = base.get('DO_ATM', True) 
        
           self.do_wave = base.get('DO_WAVE', False) 
        
           self.do_wave_bnd = base.get('DOBNDPNT_WAVE', False) 
        
           self.do_ocean = base.get('DO_OCN', False) 
        
           self.do_ice = base.get('DO_ICE', False) 
        
           self.do_aero = base.get('DO_AERO', False)

This will require reworking the get_task_names methods to key off of RUN-specific flags.

workflow/applications/applications.py

DavidHuber-NOAA · 2024-10-01T19:29:04Z

This PR now correctly generates identical CI XMLs as compared to develop (with the exception of the stage_ic and cleanup jobs which are now run on service partitions). An additional case was tested based on the C96C48_hybatmaerosnowDA case, but the APP was set to ATMA for RUN==gdas and ATM for RUN==gfs. In this case, the gfs tasks in the resultant XML did not include any aero-component jobs.

Marking ready for review.

workflow/applications/applications.py

workflow/applications/gfs_cycled.py

workflow/applications/applications.py

workflow/applications/gfs_cycled.py

workflow/applications/gfs_forecast_only.py

workflow/rocoto/gfs_tasks.py

Hercules was already correct Refs NOAA-EMC#2956

emcbot · 2024-10-17T14:31:53Z

Experiment C96C48_hybatmDA FAILED on Orion in Build# 1 in
/work2/noaa/stmp/CI/ORION/2943/RUNTESTS/EXPDIR/C96C48_hybatmDA_1975b2f3

emcbot · 2024-10-17T14:37:58Z

Experiment C96_atm3DVar FAILED on Orion in Build# 1 in
/work2/noaa/stmp/CI/ORION/2943/RUNTESTS/EXPDIR/C96_atm3DVar_1975b2f3

TerrenceMcGuinness-NOAA · 2024-10-17T16:05:51Z

mterry (orion-login-1) C96C48_hybatmDA_1975b2f3 $ pwd
/work2/noaa/stmp/CI/ORION/2943/RUNTESTS/EXPDIR/C96C48_hybatmDA_1975b2f3
mterry (orion-login-1) C96C48_hybatmDA_1975b2f3 $ rocotostat -d C96C48_hybatmDA_1975b2f3.db -w C96C48_hybatmDA_1975b2f3.xml -s
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Oct 17 2024 13:28:51    Oct 17 2024 14:30:36
202112210000      Active    Oct 17 2024 13:28:51             -          
202112210600      Active    Oct 17 2024 13:28:51             -          
mterry (orion-login-1) C96C48_hybatmDA_1975b2f3 $ /work2/noaa/stmp/CI/ORION/2943/gfs/ci/scripts/utils/rocotostat.py -d C96C48_hybatmDA_1975b2f3.db -w C96C48_hybatmDA_1975b2f3.xml
STALLED

mterry (orion-login-1) C96C48_hybatmDA_1975b2f3 $ rocotorun -v 10 -d C96C48_hybatmDA_1975b2f3.db -w C96C48_hybatmDA_1975b2f3.xml
10/17/24 11:04:43 CDT :: C96C48_hybatmDA_1975b2f3.xml :: Submitting gdas_prep using sbatch < /tmp/sbatch.in20241017-1418003-p5s697 with input
{{
#! /bin/sh
#SBATCH --job-name=C96C48_hybatmDA_1975b2f3_gdas_prep_00
#SBATCH --account=nems
#SBATCH --qos=batch
#SBATCH --partition=orion
#SBATCH -t 00:30:00
#SBATCH --nodes=2-2
#SBATCH --tasks-per-node=7
#SBATCH --cpus-per-task=1
#SBATCH --mem=192512
#SBATCH -o /work2/noaa/stmp/CI/ORION/2943/RUNTESTS/COMROOT/C96C48_hybatmDA_1975b2f3/logs/2021122100/gdas_prep.log
#SBATCH --export=NONE
#SBATCH --comment=7ab7068ceccc0ec3f97ebb4cd2c6e87f
export RUN_ENVIR='emc'
export HOMEgfs='/work2/noaa/stmp/CI/ORION/2943/gfs'
export EXPDIR='/work2/noaa/stmp/CI/ORION/2943/RUNTESTS/EXPDIR/C96C48_hybatmDA_1975b2f3'
export NET='gfs'
export RUN='gdas'
export CDATE='2021122100'
export PDY='20211221'
export cyc='00'
export COMROOT='/work2/noaa/stmp/CI/ORION/2943/RUNTESTS/COMROOT'
export DATAROOT='/work/noaa/stmp/mterry/ORION/RUNDIRS/C96C48_hybatmDA_1975b2f3/gdas.2021122100'
/work2/noaa/stmp/CI/ORION/2943/gfs/jobs/rocoto/prep.sh
}}
10/17/24 11:04:43 CDT :: C96C48_hybatmDA_1975b2f3.xml :: WARNING: job submission failed: sbatch: error: Memory specification can not be satisfied
sbatch: error: Batch job submission failed: Requested node configuration is not available
10/17/24 11:04:43 CDT :: C96C48_hybatmDA_1975b2f3.xml :: Submitting gfs_prep using sbatch < /tmp/sbatch.in20241017-1418003-pcnt7j with input
{{
#! /bin/sh
#SBATCH --job-name=C96C48_hybatmDA_1975b2f3_gfs_prep_00
#SBATCH --account=nems
#SBATCH --qos=batch
#SBATCH --partition=orion
#SBATCH -t 00:30:00
#SBATCH --nodes=2-2
#SBATCH --tasks-per-node=7
#SBATCH --cpus-per-task=1
#SBATCH --mem=192512
#SBATCH -o /work2/noaa/stmp/CI/ORION/2943/RUNTESTS/COMROOT/C96C48_hybatmDA_1975b2f3/logs/2021122100/gfs_prep.log
#SBATCH --export=NONE
#SBATCH --comment=161689d84fda74888fcee09c8a75ab8c
export RUN_ENVIR='emc'
export HOMEgfs='/work2/noaa/stmp/CI/ORION/2943/gfs'
export EXPDIR='/work2/noaa/stmp/CI/ORION/2943/RUNTESTS/EXPDIR/C96C48_hybatmDA_1975b2f3'
export NET='gfs'
export RUN='gfs'
export CDATE='2021122100'
export PDY='20211221'
export cyc='00'
export COMROOT='/work2/noaa/stmp/CI/ORION/2943/RUNTESTS/COMROOT'
export DATAROOT='/work/noaa/stmp/mterry/ORION/RUNDIRS/C96C48_hybatmDA_1975b2f3/gfs.2021122100'
/work2/noaa/stmp/CI/ORION/2943/gfs/jobs/rocoto/prep.sh
}}
10/17/24 11:04:43 CDT :: C96C48_hybatmDA_1975b2f3.xml :: WARNING: job submission failed: sbatch: error: Memory specification can not be satisfied
sbatch: error: Batch job submission failed: Requested node configuration is not available
10/17/24 11:04:43 CDT :: C96C48_hybatmDA_1975b2f3.xml :: Submission of gdas_prep failed!  sbatch: error: Memory specification can not be satisfied
sbatch: error: Batch job submission failed: Requested node configuration is not available
10/17/24 11:04:43 CDT :: C96C48_hybatmDA_1975b2f3.xml :: Submission of gfs_prep failed!  sbatch: error: Memory specification can not be satisfied
sbatch: error: Batch job submission failed: Requested node configuration is not available

DavidHuber-NOAA · 2024-10-17T16:16:08Z

Thanks @TerrenceMcGuinness-NOAA. I guess I need to readjust mem_node_max for Orion.

emcbot · 2024-10-17T16:28:35Z

Experiment C48_ATM FAILED on Orion in Build# 1 with error logs:

/work2/noaa/stmp/CI/ORION/2943/RUNTESTS/COMROOT/C48_ATM_1975b2f3/logs/2021032312/gfs_metpg2o1.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2024-10-17T16:28:42Z

Experiment C48_ATM FAILED on Orion in Build# 1 in
/work2/noaa/stmp/CI/ORION/2943/RUNTESTS/EXPDIR/C48_ATM_1975b2f3

emcbot · 2024-10-17T16:34:33Z

Experiment C96_S2SWA_gefs_replay_ics FAILED on Orion in Build# 1 in
/work2/noaa/stmp/CI/ORION/2943/RUNTESTS/EXPDIR/C96_S2SWA_gefs_replay_ics_1975b2f3

emcbot · 2024-10-17T19:47:04Z

CI Failed on Orion in Build# 1
Built and ran in directory /work2/noaa/stmp/CI/ORION/2943


Experiment C96C48_hybatmDA_1975b2f3 Terminated with 0 tasks failed and 0 dead at Thu Oct 17 09:31:44 AM CDT 2024
Experiment C96C48_hybatmDA_1975b2f3 Terminated: *STALLED*
Experiment C96_atm3DVar_1975b2f3 Terminated with 0 tasks failed and 0 dead at Thu Oct 17 09:37:53 AM CDT 2024
Experiment C96_atm3DVar_1975b2f3 Terminated: *STALLED*
Experiment C48_ATM_1975b2f3 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Thu Oct 17 11:28:20 AM CDT 2024
Experiment C48_ATM_1975b2f3 Terminated: *FAIL*
Error logs:
/work2/noaa/stmp/CI/ORION/2943/RUNTESTS/COMROOT/C48_ATM_1975b2f3/logs/2021032312/gfs_metpg2o1.log
Experiment C96_S2SWA_gefs_replay_ics_1975b2f3 Terminated with 0 tasks failed and 0 dead at Thu Oct 17 11:34:29 AM CDT 2024
Experiment C96_S2SWA_gefs_replay_ics_1975b2f3 Terminated: *STALLED*
Experiment C48_S2SW_1975b2f3 Completed 1 Cycles: *SUCCESS* at Thu Oct 17 01:45:14 PM CDT 2024
Experiment C48_S2SWA_gefs_1975b2f3 Completed 1 Cycles: *SUCCESS* at Thu Oct 17 02:46:52 PM CDT 2024

DavidHuber-NOAA · 2024-10-23T11:53:18Z

I'm having some issues with this branch after merging the interval PR. Converting to draft until I get it squared away.

DavidHuber-NOAA · 2024-10-24T19:17:01Z

All features are working again. Reopening this PR for review.

DavidHuber-NOAA added 4 commits September 19, 2024 16:34

Additions to allow APP to differ by RUN

c0102bf

Make cleanup and stage jobs run on service partitions

d987c38

Combine get_valid_runs and get_task_names

599beeb

Simplify get_task_names

eaa1049

DavidHuber-NOAA added 4 commits September 20, 2024 13:13

Make APP-specific options based on RUN

3787be7

Move _get_app_configs invokation to _source_configs

a4ab75c

Make the Application classes run-configurable

4ae0f36

Fix f-string formating

4374bad

DavidHuber-NOAA commented Sep 24, 2024

View reviewed changes

workflow/applications/applications.py Show resolved Hide resolved

DavidHuber-NOAA added 9 commits September 24, 2024 15:02

Application bug fixes

e6b1920

Update app_config options in the Tasks classes

d94b685

Fix hybvar ocn analysis task tabbing

10763cc

Merge develop branch

2f07ddb

do_extractvars bug fix

7dbf11a

Fix do_extractvars bug, remove unused modules

de57402

Fix additional do_* bugs

7253eef

Rearrange gfs_cycled task order

9e96844

Rearrange XML RUN order to match develop

8275caf

DavidHuber-NOAA force-pushed the feature/multi-app branch from 217ec77 to 8275caf Compare October 1, 2024 15:31

DavidHuber-NOAA added 2 commits October 1, 2024 10:44

Merge develop, address conflicts in gfs_cycled and gfs_tasks

60f6384

Fix merge bugs

4477e60

DavidHuber-NOAA requested review from WalterKolczynski-NOAA, aerorahul and EricSinsky-NOAA October 1, 2024 19:23

DavidHuber-NOAA marked this pull request as ready for review October 1, 2024 19:24

DavidHuber-NOAA commented Oct 2, 2024

View reviewed changes

DavidHuber-NOAA added 2 commits October 4, 2024 13:38

Increase max allocatable memory on Orion

e8a9b96

Hercules was already correct Refs NOAA-EMC#2956

Merge remote-tracking branch 'origin/develop' into feature/multi-app

e4ff015

emcbot added the CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed label Oct 17, 2024

emcbot added CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed and removed CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed labels Oct 17, 2024

DavidHuber-NOAA added 2 commits October 18, 2024 14:32

Merge remote-tracking branch 'origin/develop' into feature/multi-app

506709e

Reduce Orion available memory

76a62d0

DavidHuber-NOAA removed the CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed label Oct 18, 2024

DavidHuber-NOAA force-pushed the feature/multi-app branch from 4489259 to 76a62d0 Compare October 18, 2024 15:32

Merge in develop

8a255cc

DavidHuber-NOAA mentioned this pull request Oct 22, 2024

Replace gfs_cyc with an interval #2928

Merged

8 tasks

DavidHuber-NOAA marked this pull request as draft October 23, 2024 11:53

DavidHuber-NOAA added 2 commits October 23, 2024 14:34

Merge remote-tracking branch 'origin/develop' into feature/multi-app

69fb251

Fix interval_gfs conflicts

2d4139b

DavidHuber-NOAA added the CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress label Oct 23, 2024

Resolve additional interval_gfs conflicts

b342a67

DavidHuber-NOAA removed the CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress label Oct 24, 2024

Merge branch 'develop' into feature/multi-app

da49bdb

DavidHuber-NOAA marked this pull request as ready for review October 24, 2024 19:17

DavidHuber-NOAA added CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules and removed CI-Hercules-Passed **Bot use only** CI testing on Hercules for this PR has completed successfully CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules labels Oct 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow APP to differ between RUNs #2943

Allow APP to differ between RUNs #2943

DavidHuber-NOAA commented Sep 19, 2024 •

edited

Loading

DavidHuber-NOAA commented Sep 19, 2024

DavidHuber-NOAA commented Oct 1, 2024

emcbot commented Oct 17, 2024

emcbot commented Oct 17, 2024

TerrenceMcGuinness-NOAA commented Oct 17, 2024 •

edited

Loading

DavidHuber-NOAA commented Oct 17, 2024

emcbot commented Oct 17, 2024

emcbot commented Oct 17, 2024

emcbot commented Oct 17, 2024

emcbot commented Oct 17, 2024

DavidHuber-NOAA commented Oct 23, 2024

DavidHuber-NOAA commented Oct 24, 2024

Allow APP to differ between RUNs #2943

Are you sure you want to change the base?

Allow APP to differ between RUNs #2943

Conversation

DavidHuber-NOAA commented Sep 19, 2024 • edited Loading

Description

Type of change

Change characteristics

How has this been tested?

Checklist

DavidHuber-NOAA commented Sep 19, 2024

DavidHuber-NOAA commented Oct 1, 2024

emcbot commented Oct 17, 2024

emcbot commented Oct 17, 2024

TerrenceMcGuinness-NOAA commented Oct 17, 2024 • edited Loading

DavidHuber-NOAA commented Oct 17, 2024

emcbot commented Oct 17, 2024

emcbot commented Oct 17, 2024

emcbot commented Oct 17, 2024

emcbot commented Oct 17, 2024

DavidHuber-NOAA commented Oct 23, 2024

DavidHuber-NOAA commented Oct 24, 2024

DavidHuber-NOAA commented Sep 19, 2024 •

edited

Loading

TerrenceMcGuinness-NOAA commented Oct 17, 2024 •

edited

Loading