Naming conventions for files in cime_config/

[mnlevy1981]

Background

In recent CSEG meetings, we talked about file naming conventions to maintain consistency among all the components -- namely, should python files have the .py extension or not. The prevailing view was that python modules that are meant to be imported should have the extension, scripts that should be run from the command line should not have it, and files that currently do double duty via if __name__ == "__main__" blocks should be split into two files (a .py module and a extension-free wrapper).

One big hitch to this plan is how CIME treats cime_config/buildnml. Currently there is logic that tries to import specific functions from buildnml, and falls back to executing the script via subprocess. @billsacks and I chatted about how to modify CIME to follow these conventions, and came up with some interesting ideas we thought were worth discussing.

Potential paths forward

Assuming other groups agree that this would be a nice convention to follow, there a few possible ways to proceed.

1. The simplest fix would be to update run_sub_or_cmd() to try to import f"{filename}.py" before falling back on the current behavior of trying to import filename and running filename if the import fails. Perhaps in a later version this could collapse to "try to import filename.py or run filename", but leaving the middle check would maintain backwards compatibility for the current version.
2. Another possibility would be to allow components to make cime_config a package (i.e. include __init__.py). Then we could try to import cime_config, and fall back to the current behavior in CIME6 (again, perhaps cutting out the import filename option in future versions). This could potentially clean up some python code in components -- one example is that POP has a python layer that sits between buildnml; if these functions are included in the cime_config package it is easier to import them into buildnml.
3. A third possibility would be to enforce a rule that each component has cime_config/${COMPNAME}/__init__.py; this would let us more cleanly separate the python code in cime_config from the XML and the test suite files. It might also let us clean up import statements since there is no longer a namespace conflict every time you import a new buildnml (or cime_config in the above option)

Questions

1. Does it make sense to make buildnml a wrapper script that imports buildnml.py?
2. If so, are any of the above paths the right answer for how CIME should make use of this renamed module?

In case it changes how you answer the above questions, I'll note that I am happy to take lead on implementing whatever solution results from this issue ticket.

[jedwards4b]

@mnlevy1981 thanks for taking this on - I'm interested in seeing what a prototype of each of the options would look like. Maybe start with option 3 (or another if others feel strongly about it).

[billsacks]

I would also lean slightly towards (3) (then (2) then (1)). But in thinking about this a little more, I'd prefer if the package name were more specific than ${COMPNAME}. This is mostly for selfish ctsm-centric reasons, because we already have a python package named ctsm (in https://github.com/ESCOMP/CTSM/tree/master/python), but I could imagine the same being true for other components. I think that, even if we had a true package under cime_config, we'd want to keep our core ctsm python in python/ctsm because it's a more intuitive place for people to find it. So then we could have import conflicts if we have two packages named ctsm in different places.

So maybe option (3) but a path like cime_config/${COMPNAME}_cime_config_py/__init__.py? (I'm not sure about the final _py after ${COMPNAME}_cime_config: I added it to make it clear to people browsing the code that this is the directory where you can find python modules, but I don't feel strongly about it.)

[rljacob]

Other suggestions from telecon:
${COMPNAME}_cime_py
${COMPNAME}_cimepck_py

NOT ${COMPNAME}_py because another model may use that in some other context.

We all liked option 3 and just need to settle on a name.

[billsacks]

I think that the ideas proposed here will also make for a cleaner method of importing system test modules that are defined in components. For example, see this comment that I wrote a while ago:

https://github.com/ESCOMP/CTSM/blob/09ef76bddcd4681bff4cec6b1043b1f8171f8722/cime_config/SystemTests/lii2finidatareas.py#L44-L57