Squashed 'manage_externals/' changes from 1926530f..38bcc0a8

38bcc0a8 Merge pull request ESCOMP#201 from jedwards4b/partial_match b4466a5a remove debug print statement c3cf3ec3 fix issue with partial branch match 7b6d92ef Merge pull request ESCOMP#198 from johnpaulalex/gitdir 927ce3a9 Merge pull request ESCOMP#197 from johnpaulalex/testpath a04f1148 Merge pull request ESCOMP#196 from johnpaulalex/readmod d9c14bf2 Change the rest of the methods to use -C. Still some usage of getcwd in test_unit_repository_git. 332b1064 Fix incorrect logged path of checkout_externals in test_sys_checkout: it was basically the parent of the current directory, which varies throughout the test. (it called abspath with '{0}/../../', which adds arbitrary and not-interpolated subdir '{0}' to the path, then removes it and removes one more level). 932a7499 Remove printlog from read_gitmodules_file since read_externals_description_file() already has a nearly-the-same printlog (but add it to the other caller). 5d13719e Merge pull request ESCOMP#195 from johnpaulalex/check_repo 42339544 Update utest to mock _git_remote_verbose in a new way, since it is now called via the GitRepository class rather than on the specific GitRepository instance. d7a42ae9 Check that desired repo was actually checked out. 71596bbc Merge pull request ESCOMP#194 from johnpaulalex/manic2 4c96e824 Make the MANIC_TEST_BARE_REPO_ROOT env var special - give it a constant for easy tracking, and automatically tear it down after each test. 259bfc04 test_sys_checkout: use actual paths in on-the-fly configs rather than MANIC_TEST_BARE_REPO_ROOT env var. This will make it easier to test (in the near future) that checkout_externals actually checked out the desired repo dir. 557bbd6e Merge pull request ESCOMP#193 from johnpaulalex/manic 5314eede Remove MANIC_TEST_TMP_REPO_ROOT environment variable in favor of module-level variable. 345fc1e1 Merge pull request ESCOMP#191 from johnpaulalex/test_doc12 2117b843 test_sys_checkout: verify that basic by-tag/branch/hash tests actually take us to the correct git tag/branch/hash. 94d6e5f2 Merge pull request ESCOMP#190 from johnpaulalex/test_doc11 3ff33a6a Inline local-path-creation methods 47dea7f6 Merge pull request ESCOMP#189 from johnpaulalex/test_doc10 9ea75cbf Grab-bag of renamings: Remove redundant _NAME from repo constants, and consistently add _REPO suffix (This causes the majority of diffs). c0c847ec Merge pull request ESCOMP#188 from johnpaulalex/test_doc9 2dd5ce0f test_sys_checkout.py: only check for correct 'required' or 'optional' state in the test that exercises required vs optional behavior. Removed a lot of boilerplate. eb308598 Merge pull request ESCOMP#187 from johnpaulalex/test_doc8 1832e1f8 test_sys_checkout: Simplify many tests to only use a single external. 8689d61e Merge pull request ESCOMP#186 from johnpaulalex/test_doc7 fbee4253 Grab bag of test_sys_checkout cleanups: Doc inside of each test more clearly/consistently. TestSysCheckoutSVN didn’t get the inlining-of-helper-methods treatment, now it has that. Move various standalone repo helper methods (like create_branch) into a RepoUtils class. README.md was missing newlines when rendered as markdown. Doc the return value of checkout.main Fix test_container_exclude_component - it was looking for the wrong key (which is never present); now it looks for the correct key. f0ed44a6 Merge pull request ESCOMP#185 from johnpaulalex/test_doc6 a3d59f5f Merge pull request ESCOMP#184 from johnpaulalex/test_doc5 5329c8ba test_sys_checkout: Inline config generation functions that are only called once. 464f2c7a test_sys_checkout: Inline another layer (per-config-file checks). Rename the 4 methods that are used multiple times, to reflect what they do rather than what they're called. 8872c0df Merge pull request ESCOMP#183 from johnpaulalex/doc_test4 c045335f Merge pull request ESCOMP#182 from johnpaulalex/doc_test3 c583b956 Merge pull request ESCOMP#181 from johnpaulalex/doc_test2 e01cfe27 test_sys_checkout: less confusing handling of return values from checkout_externals. Specifically, when doing a checkout, don't return tree_status from _before_ the checkout. Make a new wrapper to call checkout_externals a second time, to calculate the new status after a checkout (very frequent pattern). 23286818 test_sys_checkout: Remove another layer (which generates test component names) c3717b6b Merge pull request ESCOMP#180 from johnpaulalex/doc_test 36d7a443 test_sys_checkout.py: remove one layer of functions (that check for local status enums). No-op. 2c4584bf More documentation about tests: * contents of test repositories (n a new README.md) * various constants in test_sys_checkout.py that point to those contents, and terminology like container/simple/mixed. * in each test method, the scenarios being tested. * The coupling between test methods. 55e74bd0 Merge pull request ESCOMP#179 from johnpaulalex/circ 66be8429 Remove circular dependency by making _External stop doing tricky things with sourcetrees. 82d3b247 Merge pull request ESCOMP#178 from johnpaulalex/test_doc 3223f49e Additional documentation of system tests - global variables, method descriptions. 45b7c01c Merge pull request ESCOMP#177 from jedwards4b/git_workflow ace90b2c try setting credentials this way f4d6aa93 try setting credentials this way 1d61a694 use this to set git credentials 7f9d330e use this to set git credentials 5ac731b8 add tmate code 836847be get git workflow working dcd462d7 Merge pull request ESCOMP#176 from jedwards4b/add_github_testing 2d2479e9 Merge pull request ESCOMP#175 from johnpaulalex/fix 711a53fd add github testing of prs and automatic tagging of main cfe0f888 fix typos 5665d614 Fix broken checkout behavior introduced by PR ESCOMP#172. 27909e25 Merge pull request ESCOMP#173 from johnpaulalex/readall 00ad0440 Further tiny refactorings and docs of checkout API (no-op). Remove unused load_all param in _External.checkout(). Rename _External.checkout_externals() to checkout_subexternals(), to remove the ambiguity about whether the main external pointed to by the _External is itelf checked out (it is not) Clarify load_all documentation - it’s always recursive, but applies different criteria at each level. Rename variables in checkout.py (e.g. ext_description) to match the equivalent code in sourcetree.py. 2ea3d1a3 Merge pull request ESCOMP#172 from johnpaulalex/fixit 43bf8092 Merge pull request ESCOMP#171 from johnpaulalex/docstatus e6aa7d21 Merge pull request ESCOMP#170 from johnpaulalex/printdir adbd7155 On checkout, refresh locally installed optional packages regardless of whether -o is passed in. add07459 Comment tweaks, and fix 'ppath' typo 696527cb Document the format of various status dictionaries, and the various paths and path components within an _External. c677b940 When processing an external, print out its path in addition to the base filename (to disambiguate all the externals.cfg's) 975d7fd5 Merge pull request ESCOMP#169 from johnpaulalex/docfix_branch 09709e36 Document _Externals.status(). The original comment was apparently copy-pasted from checkout(). 1d880e09 Merge pull request ESCOMP#167 from billsacks/fix_svn_on_windows 3510da84 Tweak a unit test to improve coverage eb7fc136 Handle the possibility that the URL already ends with '/' 02ea87e3 Fix svn URLs on Windows b1c02ab5 Merge pull request ESCOMP#165 from gold2718/doc_fix 9f4be8c7 Add documentation about externals = None feature a3b3a037 Merge pull request ESCOMP#162 from ESMCI/fischer/python3 d4f1b1e8 Change shebang lines to python3 2fd941ab Merge pull request ESCOMP#158 from billsacks/modified_solution de08dc2e Add another option for when an external is in a modified state e954582d Merge pull request ESCOMP#156 from billsacks/onbranch_show_hash 952e44d5 Change output: put tag/hash before branch name 10288430 Fix pre-existing pylint issues 01b13f78 When on a branch, show tag/hash, too 39ad5326 Merge pull request ESCOMP#150 from gold2718/fix_combo_config 75f8f02f Merge pull request ESCOMP#152 from jedwards4b/sort_by_local_path 42687bd5 remove commented code 29e26af8 fix pylint issues 7c9f3c61 add a test for nested repo checkout 75c5353d fix spacing 24a3726a improve sorting, checkout externals with each comp 29f45b08 remove py2 test and fix super call 880a4e76 remove decode 1c53be85 no need for set call 36c56dba simplier fix for issue dc67cc68 simpler solution b32c6fca fix to allow submodule name different from path 5b5e1c2b Merge pull request ESCOMP#144 from billsacks/improve_errmsg c983863c Add another option for dealing with modified externals 59ce252c Add some details to the error message when externals are modified be5a1a4d Merge pull request ESCOMP#143 from jedwards4b/add_exclude 2aa014a1 fix lint issue 49cd5e89 fix lint issues 418173ff Added tests for ExternalsDescriptionDict afab352c fix lint issue be85b7d1 fix the test a580a570 push test d4371086 add a test 21affe33 fix formatting issue 72e6b64a add an exclude option c33a3bd2 Merge pull request ESCOMP#139 from jedwards4b/ignore_branch_prop b124a9af ignore this silently, we use a hash so it does not matter fde04e4d Merge pull request ESCOMP#138 from billsacks/add_python38_tests 37e4c4a5 Do not update dictionary in-place in loop 7e8474bb Remove testing on mac os 7f41c563 Fix pylint issue 3065b0d6 Add travis-ci tests with python3.7 and python3.8 34fbf556 Add support for git sparse checkout 6c6ef9fe Fix pylint errors 6a659ad3 Added test for sparse checkout and updated documentation 14432439 Support for git sparsecheckout via read-tree. a48558d8 Merge pull request ESCOMP#119 from gold2718/submodules f72ffe7f Do not try git submodule update if no .gitmodules file (git bug) 804e0afb Fix a pylint error 45aef95e Addressed review concerns 7da50314 New capability to use git submodule information to checkout externals git-subtree-dir: manage_externals git-subtree-split: 38bcc0a8c9fb23e209e5f3d58344bedb06ab5018
nmizukami · Nov 9, 2023 · ef36fe7 · ef36fe7
1 parent c610be4
commit ef36fe7
Show file tree

Hide file tree

Showing 32 changed files with 2,671 additions and 1,563 deletions.
diff --git a/.github/workflows/bumpversion.yml b/.github/workflows/bumpversion.yml
@@ -0,0 +1,19 @@
+name: Bump version
+on:
+  push:
+    branches:
+      - main
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - name: Bump version and push tag
+        id: tag_version
+        uses: mathieudutour/github-tag-action@v5.5
+        with:
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          create_annotated_tag: true
+          default_bump: patch
+          dry_run: false
+          tag_prefix: manic-
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
@@ -0,0 +1,30 @@
+# This is a workflow to compile the cmeps source without cime
+name: Test Manic
+
+# Controls when the action will run. Triggers the workflow on push or pull request
+# events but only for the master branch
+on:
+  push:
+    branches: [ main ]
+  pull_request:
+    branches: [ main ]
+
+# A workflow run is made up of one or more jobs that can run sequentially or in parallel
+jobs:
+  test-manic:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - name: Test Manic
+        run: |
+          pushd test
+          git config --global user.email "devnull@example.com"
+          git config --global user.name "GITHUB tester"
+          git config --global protocol.file.allow always
+          make utest
+          make stest
+          popd
+
+      - name: Setup tmate session
+        if: ${{ failure() }}
+        uses: mxschmitt/action-tmate@v3
diff --git a/.gitignore b/.gitignore
@@ -12,3 +12,6 @@ components/
 
 # generated python files
 *.pyc
+
+# test tmp file
+test/tmp
diff --git a/.travis.yml b/.travis.yml
@@ -1,25 +1,11 @@
-# NOTE(bja, 2017-11) travis-ci dosen't support python language builds
-#  on mac os. As a work around, we use built-in python on linux, and
-#  declare osx a 'generic' language, and create our own python env.
-
 language: python
 os: linux
-python: 
-  - "2.7"
+python:
   - "3.4"
   - "3.5"
   - "3.6"
-matrix:
-  include:
-  - os: osx
-    language: generic
-    before_install:
-      # NOTE(bja, 2017-11) update is slow, 2.7.12 installed by default, good enough!
-      # - brew update
-      # - brew outdated python2 || brew upgrade python2
-      - pip install virtualenv
-      - virtualenv env -p python2
-      - source env/bin/activate
+  - "3.7"
+  - "3.8"
 install:
   - pip install -r test/requirements.txt
 before_script:

diff --git a/README.md b/README.md
@@ -85,7 +85,7 @@ The root of the source tree will be referred to as `${SRC_ROOT}` below.
     description file:
 
         $ cd ${SRC_ROOT}
-        $ ./manage_externals/checkout_externals --excernals my-externals.cfg
+        $ ./manage_externals/checkout_externals --externals my-externals.cfg
 
   * Status summary of the repositories managed by checkout_externals:
 
@@ -201,6 +201,26 @@ The root of the source tree will be referred to as `${SRC_ROOT}` below.
     externals description file pointed 'useful_library/sub-xternals.cfg',
     Then the main 'externals' field in the top level repo should point to
     'sub-externals.cfg'.
+    Note that by default, `checkout_externals` will clone an external's
+    submodules. As a special case, the entry, `externals = None`, will
+    prevent this behavior. For more control over which externals are
+    checked out, create an externals file (and see the `from_submodule`
+    configuration entry below).
+
+  * from_submodule (True / False) : used to pull the repo_url, local_path,
+    and hash properties for this external from the .gitmodules file in
+    this repository. Note that the section name (the entry in square
+    brackets) must match the name in the .gitmodules file.
+    If from_submodule is True, the protocol must be git and no repo_url,
+    local_path, hash, branch, or tag entries are allowed.
+    Default: False
+
+  * sparse (string) : used to control a sparse checkout. This optional
+    entry should point to a filename (path relative to local_path) that
+    contains instructions on which repository paths to include (or
+    exclude) from the working tree.
+    See the "SPARSE CHECKOUT" section of https://git-scm.com/docs/git-read-tree
+    Default: sparse checkout is disabled
 
   * Lines begining with '#' or ';' are comments and will be ignored.
 

diff --git a/checkout_externals b/checkout_externals
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 """Main driver wrapper around the manic/checkout utility.
 

diff --git a/manic/checkout.py b/manic/checkout.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 """
 Tool to assemble repositories represented in a model-description file.
@@ -227,6 +227,27 @@ def commandline_arguments(args=None):
     Now, %(prog)s will process Externals.cfg and also process
     Externals_LIBX.cfg as if it was a sub-external.
 
+    Note that by default, checkout_externals will clone an external's
+    submodules. As a special case, the entry, "externals = None", will
+    prevent this behavior. For more control over which externals are
+    checked out, create an externals file (and see the from_submodule
+    configuration entry below).
+
+  * from_submodule (True / False) : used to pull the repo_url, local_path,
+    and hash properties for this external from the .gitmodules file in
+    this repository. Note that the section name (the entry in square
+    brackets) must match the name in the .gitmodules file.
+    If from_submodule is True, the protocol must be git and no repo_url,
+    local_path, hash, branch, or tag entries are allowed.
+    Default: False
+
+  * sparse (string) : used to control a sparse checkout. This optional
+    entry should point to a filename (path relative to local_path) that
+    contains instructions on which repository paths to include (or
+    exclude) from the working tree.
+    See the "SPARSE CHECKOUT" section of https://git-scm.com/docs/git-read-tree
+    Default: sparse checkout is disabled
+
   * Lines beginning with '#' or ';' are comments and will be ignored.
 
 # Obtaining this tool, reporting issues, etc.
@@ -264,6 +285,9 @@ def commandline_arguments(args=None):
                         help='The externals description filename. '
                         'Default: %(default)s.')
 
+    parser.add_argument('-x', '--exclude', nargs='*',
+                        help='Component(s) listed in the externals file which should be ignored.')
+
     parser.add_argument('-o', '--optional', action='store_true', default=False,
                         help='By default only the required externals '
                         'are checked out. This flag will also checkout the '
@@ -314,7 +338,34 @@ def commandline_arguments(args=None):
         options = parser.parse_args()
     return options
 
-
+def _dirty_local_repo_msg(program_name, config_file):
+    return """The external repositories labeled with 'M' above are not in a clean state.
+The following are four options for how to proceed:
+(1) Go into each external that is not in a clean state and issue either a 'git status' or
+    an 'svn status' command (depending on whether the external is managed by git or
+    svn). Either revert or commit your changes so that all externals are in a clean
+    state. (To revert changes in git, follow the instructions given when you run 'git
+    status'.) (Note, though, that it is okay to have untracked files in your working
+    directory.) Then rerun {program_name}.
+(2) Alternatively, you do not have to rely on {program_name}. Instead, you can manually
+    update out-of-sync externals (labeled with 's' above) as described in the
+    configuration file {config_file}. (For example, run 'git fetch' and 'git checkout'
+    commands to checkout the appropriate tags for each external, as given in
+    {config_file}.)
+(3) You can also use {program_name} to manage most, but not all externals: You can specify
+    one or more externals to ignore using the '-x' or '--exclude' argument to
+    {program_name}. Excluding externals labeled with 'M' will allow {program_name} to
+    update the other, non-excluded externals.
+(4) As a last resort, if you are confident that there is no work that needs to be saved
+    from a given external, you can remove that external (via "rm -rf [directory]") and
+    then rerun the {program_name} tool. This option is mainly useful as a workaround for
+    issues with this tool (such as https://github.com/ESMCI/manage_externals/issues/157).
+The external repositories labeled with '?' above are not under version
+control using the expected protocol. If you are sure you want to switch
+protocols, and you don't have any work you need to save from this
+directory, then run "rm -rf [directory]" before rerunning the
+{program_name} tool.
+""".format(program_name=program_name, config_file=config_file)
 # ---------------------------------------------------------------------
 #
 # main
@@ -327,9 +378,9 @@ def main(args):
     the --all option is passed.
 
     Returns a tuple (overall_status, tree_status). overall_status is 0
-    on success, non-zero on failure. tree_status gives the full status
-    *before* executing the checkout command - i.e., the status that it
-    used to determine if it's safe to proceed with the checkout.
+    on success, non-zero on failure. tree_status is a dict mapping local path
+    to ExternalStatus -- if no checkout is happening. If checkout is happening, tree_status
+    is None.
     """
     if args.do_logging:
         logging.basicConfig(filename=LOG_FILE_NAME,
@@ -345,64 +396,50 @@ def main(args):
         load_all = True
 
     root_dir = os.path.abspath(os.getcwd())
-    external_data = read_externals_description_file(root_dir, args.externals)
-    external = create_externals_description(
-        external_data, components=args.components)
+    model_data = read_externals_description_file(root_dir, args.externals)
+    ext_description = create_externals_description(
+        model_data, components=args.components, exclude=args.exclude)
 
     for comp in args.components:
-        if comp not in external.keys():
+        if comp not in ext_description.keys():
+            # Note we can't print out the list of found externals because
+            # they were filtered in create_externals_description above.
             fatal_error(
                 "No component {} found in {}".format(
                     comp, args.externals))
 
-    source_tree = SourceTree(root_dir, external, svn_ignore_ancestry=args.svn_ignore_ancestry)
-    printlog('Checking status of externals: ', end='')
-    tree_status = source_tree.status()
+    source_tree = SourceTree(root_dir, ext_description, svn_ignore_ancestry=args.svn_ignore_ancestry)
+    if args.components:
+        components_str = 'specified components'
+    else:
+        components_str = 'required & optional components'
+    printlog('Checking local status of ' + components_str + ': ', end='')
+    tree_status = source_tree.status(print_progress=True)
     printlog('')
 
     if args.status:
         # user requested status-only
-        for comp in sorted(tree_status.keys()):
+        for comp in sorted(tree_status):
             tree_status[comp].log_status_message(args.verbose)
     else:
         # checkout / update the external repositories.
         safe_to_update = check_safe_to_update_repos(tree_status)
         if not safe_to_update:
             # print status
-            for comp in sorted(tree_status.keys()):
+            for comp in sorted(tree_status):
                 tree_status[comp].log_status_message(args.verbose)
             # exit gracefully
-            msg = """The external repositories labeled with 'M' above are not in a clean state.
-
-The following are two options for how to proceed:
-
-(1) Go into each external that is not in a clean state and issue either
-    an 'svn status' or a 'git status' command. Either revert or commit
-    your changes so that all externals are in a clean state. (Note,
-    though, that it is okay to have untracked files in your working
-    directory.) Then rerun {program_name}.
-
-(2) Alternatively, you do not have to rely on {program_name}. Instead, you
-    can manually update out-of-sync externals (labeled with 's' above)
-    as described in the configuration file {config_file}.
-
-
-The external repositories labeled with '?' above are not under version
-control using the expected protocol. If you are sure you want to switch
-protocols, and you don't have any work you need to save from this
-directory, then run "rm -rf [directory]" before re-running the
-checkout_externals tool.
-""".format(program_name=program_name, config_file=args.externals)
-
             printlog('-' * 70)
-            printlog(msg)
+            printlog(_dirty_local_repo_msg(program_name, args.externals))
             printlog('-' * 70)
         else:
             if not args.components:
                 source_tree.checkout(args.verbose, load_all)
             for comp in args.components:
                 source_tree.checkout(args.verbose, load_all, load_comp=comp)
             printlog('')
+            # New tree status is unknown, don't return anything.
+            tree_status = None
 
     logging.info('%s completed without exceptions.', program_name)
     # NOTE(bja, 2017-11) tree status is used by the systems tests